A method for predicting controller voice fatigue based on remote tower scenario

Abstract

The present invention relates to the field of voice data processing technology, and discloses a method for predicting controller voice fatigue based on a remote tower scenario. The method comprises the following steps: collecting audio samples containing controller voices from remote tower call data and performing denoising to obtain denoised voice signals; labeling the voice signals for fatigue status; extracting rhythmic, timbre, and spectral features from the labeled voice signals and fusing them into comprehensive features; training a Bi-LSTM-GRU model based on the extracted comprehensive features to output fatigue prediction results; and collecting real-time voice data from the controller, inputting the real-time voice data into the trained Bi-LSTM-GRU model to predict the controller's fatigue status. The present invention aims to improve the accuracy of predicting controller voice fatigue status.

Description

Technical Field

[0001] The present invention relates to the technical field of voice data processing, and in particular to a method for predicting controller voice fatigue based on a remote tower scenario. Background Art

[0002] Voice fatigue detection technology determines speaker fatigue by analyzing characteristic changes in voice signals. With the increasing demand for fatigue detection, researchers have proposed a variety of fatigue detection methods based on machine learning and deep learning. Currently, methods for predicting controller voice fatigue primarily rely on feature extraction of controller voice signals and analysis using traditional machine learning models. Traditional methods typically employ classic algorithms such as support vector machines (SVMs) and decision trees. While these methods can achieve good results in some simple scenarios, when faced with complex voice signals, the feature selection and model training processes are susceptible to noise and data imbalance, resulting in low prediction accuracy and difficulty adapting to the changing fatigue states of controllers in high-stress, high-intensity work environments.

[0003] Therefore, existing methods for predicting controller voice fatigue suffer from low accuracy, poor robustness, and poor real-time performance when faced with complex working environments, dynamic fatigue states, and data imbalance. Therefore, a new prediction method is urgently needed to improve the accuracy, robustness, and real-time performance of controller fatigue prediction, especially in high-stress and complex environments. Summary of the Invention

[0004] The purpose of the present invention is to improve the prediction accuracy of the controller's voice fatigue state and provide a controller voice fatigue prediction method based on a remote tower scenario.

[0005] In order to achieve the above-mentioned purpose of the invention, the embodiment of the present invention provides the following technical solution: a method for predicting controller voice fatigue based on a remote tower scenario, comprising the following steps:

[0006] Step 1: Collect audio samples containing controller voices from remote tower call data and perform denoising to obtain denoised voice signals; then label the voice signals with fatigue status.

[0007] Step 2: extract prosody features, sound quality features, and spectral features from several annotated speech signals and fuse them into comprehensive features;

[0008] Step 3: Training the Bi-LSTM-GRU model based on the extracted comprehensive features to output fatigue prediction results; the Bi-LSTM-GRU model includes an input layer, a backward layer, a forward layer, and an output layer, wherein the input layer transmits the comprehensive features of the time step to the backward layer and the forward layer respectively, the backward layer processes the reverse time series at the time step, the forward layer processes the forward time series at the time step, and the output layer splices the outputs in the forward and backward directions;

[0009] Step 4: Collect the real-time voice data of the controller and input the real-time voice data into the trained Bi-LSTM-GRU model to predict the controller's fatigue status.

[0010] Compared with the existing technology, the present invention effectively overcomes the technical problems in the existing controller voice fatigue prediction method by introducing innovative technical means such as an improved Bi-LSTM-GRU model (bidirectional long short-term memory network), subjective and objective feature fusion, and dynamic threshold adjustment mechanism, and produces significant technical effects, which are specifically reflected in the following aspects.

[0011] (1) Overcoming the feature dependency problem and improving the accuracy of fatigue prediction, the present invention combines multi-dimensional features such as rhythm, sound quality and spectrum, so that the prediction of fatigue status no longer depends on a single static feature. The comprehensive use of such multi-dimensional features greatly improves the performance of the model in complex environments, especially in high-noise and variable remote tower control working environments, effectively avoiding the interference of external factors on feature data and improving the accuracy of fatigue prediction. At the same time, the introduction of subjective fatigue scales (such as the Samn-Perelli 7-level fatigue scale and the SART scale) combined with objective reaction time (such as the PVT test) can effectively and accurately mark the fatigue and non-fatigue of controllers after standardization. Through standardized subjective and objective scoring, the present invention can capture subtle changes in fatigue earlier and more accurately, especially when the controller is under high-pressure and high-intensity work conditions. This innovative method effectively solves the problem of misjudgment of fatigue status by traditional methods, especially avoiding missing subtle changes in the early stages of fatigue, providing timely and accurate fatigue status prediction, and ensuring flight safety and the work health of controllers.

[0012] (2) This paper overcomes the limitations of traditional machine learning methods and improves the adaptability and efficiency of the system. The present invention adopts a Bi-LSTM-GRU model based on deep learning. Compared with traditional methods such as support vector machines (SVM) and decision trees, the Bi-LSTM-GRU model can better process time series data and capture the dynamic changes in the controller's fatigue state. Through the structure of forward LSTM and reverse GRU, the model not only considers historical information, but also makes full use of future information, greatly improving the model's sensitivity to changes in fatigue state. In addition, the Bi-LSTM-GRU model has a strong feature learning ability, avoiding the problem of relying on manual feature selection in traditional methods, and improving the system's adaptability to fatigue changes under different working conditions. Especially in high-pressure environments, it can accurately capture subtle changes in fatigue, avoiding the failure of warning caused by the traditional method's over-reliance on static features.

[0013] (3) The problem of static characteristics and progressive fatigue is solved, and fatigue prediction is achieved. By using the Bi-LSTM-GRU model to perform deep learning on time series features, the present invention can effectively identify the gradual changes in the fatigue state of the controller, especially the progressive characteristics of fatigue after long-term work. Unlike traditional methods that rely on static characteristics, this model can capture subtle changes in fatigue during long-term, high-intensity work and identify fatigue signals in advance, thereby effectively avoiding the omission of early fatigue by traditional methods, ensuring the timely response of the early warning system, and providing more accurate predictions for flight safety.

[0014] (4) Overcoming the high cost and data imbalance problems and improving the robustness of the model, the present invention utilizes the self-learning capability of the deep learning model. This allows the present invention to acquire controller voice data in real time for training and prediction without relying on a large amount of manually labeled data, thus reducing the reliance on manually labeled data during system training. Furthermore, the model can adaptively adjust to address data imbalance during training, improving prediction accuracy and robustness in the absence of fatigue samples. This enables the present invention to adapt to the working conditions and environments of different controllers in practical applications, maintaining a high level of fatigue prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 Flow chart of the method of the present invention;

[0017] Figure 2 This is a schematic diagram of the Bi-LSTM-GRU network structure of the present invention. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0019] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance, or implying any such actual relationship or order between these entities or operations. In addition, the terms "connected" and "connected" can refer to direct connection between elements or indirect connection via other elements.

[0020] The present invention is achieved through the following technical solutions: Figure 1 As shown, a controller voice fatigue prediction method based on a remote tower scenario includes the following steps:

[0021] Step 1: Collect audio samples containing controller voices from remote tower call data and perform denoising to obtain denoised voice signals; then label the voice signals with fatigue status.

[0022] Audio samples containing controller speech were collected from real remote tower conversation data. These samples covered various working conditions, such as different work hours (morning and night shifts), and varying ambient noise levels to ensure the broad applicability of the analysis results. The samples also included both fatigued and non-fatigued controller speech. Depending on the specific circumstances, audio samples were collected from different airports and different controllers to ensure the diversity of the audio samples.

[0023] Because controllers in remote towers handle not just a single airport but multiple integrated airports, the complexities of control and command make the collected voice data noisy, hindering subsequent fatigue prediction. Therefore, the collected audio samples are denoised using noise suppression algorithms, such as spectral subtraction, to remove background noise and interfering sounds, ensuring voice signal clarity. Volume and sound quality are adjusted to uniformly maintain the audio level, minimizing feature extraction errors caused by volume variations. Silence segments are then removed from the voice signal to eliminate segments without valid speech content, ensuring that only the controller's voice is included in subsequent analysis.

[0024] In detail, the time-frequency domain representation of the original speech signal in the collected audio samples is,

[0025]

[0026] Where f is frequency; t is time; x(t) is the original speech signal; X(f,t) is the complex value of the original speech signal at time t and frequency f; w(t-t0) is the window function; t0 is the center time of the current window; and F is the Fourier transform operation.

[0027] In spectral subtraction, the power spectrum is used to reduce the noise of the signal. Assume that the noise spectrum estimate is , denoising is performed by spectral subtraction to obtain the denoised speech signal,

[0028]

[0029] Where |X(f,t)| 2 is the square of the modulus of X(f,t), that is, the power spectrum of the original speech signal; is the power spectrum of the noise estimate; S(f,t) is the power spectrum of the denoised speech signal. The max operation is used to ensure that the spectrum value does not become negative, avoiding unreasonable results.

[0030] By performing an inverse short-time Fourier transform on the power spectrum S(f,t) of the denoised speech signal, the corresponding time domain signal can be obtained.

[0031]

[0032] in, represents the time domain signal corresponding to S(f,t); F -1 Represents an inverse short-time Fourier transform operation.

[0033] The de-noised speech signal is annotated, divided into several segments, and each segment is labeled with its fatigue status (fatigue, non-fatigue). The fatigue status annotation takes into account both the controller's subjective fatigue perception and objective fatigue testing.

[0034] Subjective fatigue perception assessment, using a subjective fatigue scale (such as the Samn-Perelli 7-level fatigue scale shown in Table 1 and the SART scale shown in Table 2) to annotate each speech sample to evaluate the controller's subjective fatigue score.

[0035] Table 1 Samn-Perelli 7-level fatigue scale

[0036]

[0037] Table 2 SART scale

[0038]

[0039] Objective fatigue test assessment, PVT (Psychomotor Vigilance Test) is a psychological test commonly used to evaluate individual reaction time. It is usually used to measure an individual's alertness, attention and responsiveness over a certain period of time. The PVT test tests an individual's reaction time (RT) by presenting visual or auditory stimuli. It refers to the time interval from the presentation of the stimulus (the appearance of a target on the screen) to the subject's response (such as pressing a button). The subject is scored according to the reaction time scoring table shown in Table 3. It is usually measured in milliseconds (ms) and is calculated as follows:

[0040]

[0041] Wherein, RT represents the reaction time of the subject obtained through the PVT test; t response represents the time point when the subject responds; t stimulus Indicates the time point of stimulus presentation.

[0042] Table 3 Reaction time score table

[0043]

[0044] Calculate the comprehensive fatigue score,

[0045]

[0046] Among them, Fatigue score represents the comprehensive fatigue score; SP score SART represents the Samn-Perelli 7-level fatigue rating score;score represents the SART scale score; RT score Indicates the PVT test score.

[0047] When the comprehensive fatigue score Fatigue score When the score exceeds 6, the subject is judged to be in a state of fatigue. score The speech signal is marked. If the score exceeds 6 points, it is marked as fatigue, otherwise it is marked as non-fatigue.

[0048] Step 2: Extract rhythm features, sound quality features, and spectral features from several annotated speech signals and fuse them into comprehensive features.

[0049] A corpus of annotated speech signals forms the foundation for subsequent feature extraction and analysis. It must contain sufficient samples to cover a wide range of possible scenarios, including but not limited to individual differences among controllers (e.g., gender, age, and experience level), different work shifts (e.g., daytime, nighttime, and continuous duty), different control tasks (e.g., arrival, departure, tower, and area control), varying workloads (e.g., low, medium, and high traffic flow control), and varying scenario complexity (e.g., single-airport, multi-airport joint command, and emergency response). Each speech sample in the corpus should include both the speech signal and the corresponding annotations.

[0050] A variety of features are extracted from the speech signals in the corpus, including rhythmic features, timbre features, and spectral features. These features can reflect various aspects of the controller's voice and are crucial for identifying fatigue status.

[0051] 1. Prosodic features include speaking rate, pause frequency, and pause duration.

[0052] A. Speech Rate (SR). Speech rate is usually measured in units of pronunciation per second (such as syllables or words). The calculation formula is:

[0053]

[0054] Among them, SR represents the speaking speed; N s Indicates the total number of syllables pronounced in the speech sample; T d Indicates the duration of the speech sample segment.

[0055] B. Pause Frequency (PF), Pause frequency refers to the number of pauses in a speech sample. It can be calculated by detecting the silent segments in the speech. The calculation formula is:

[0056]

[0057] Among them, PF represents the pause frequency; N p represents the number of pauses in the speech sample; T d Indicates the duration of the speech sample segment.

[0058] C. Pause Duration (PD), the pause duration is expressed as the average duration of each pause, and the calculation formula is:

[0059]

[0060] Among them, PD represents the pause duration; Indicates the duration of each pause; i represents the index of the i-th pause, i=1,2,...,N p .

[0061] The calculation formula of rhythmic features is:

[0062]

[0063] Among them, [SR, PF, PD] means splicing the speech rate, pause frequency and pause duration according to the time dimension; F prosody A vector representing rhythmic features. Since SR, PF, and PD are all one-dimensional values, concatenating them forms a 1*3 row vector.

[0064] 2. Sound quality features reflect the clarity, stability, and other factors related to pronunciation quality. Sound quality features include sound pressure level and fundamental frequency.

[0065] A. Sound Pressure Level (SPL) measures the loudness of speech and is calculated as follows:

[0066]

[0067] Where SPL represents the sound pressure level; p represents the current sound pressure; p0 represents the reference sound pressure, which is usually 20×10 -6 Pa.

[0068] B. Fundamental Frequency (F0), the fundamental frequency reflects the pitch of the sound, and the calculation formula is,

[0069]

[0070] Among them, F0 represents the basic frequency; N cycles Indicates the number of vibrations in a cycle; N period Indicates the cycle duration.

[0071] The calculation formula for sound quality features is:

[0072]

[0073] Among them, [SPL,F0] represents the concatenation of sound pressure level and fundamental frequency in the time dimension; F quality A vector representing sound quality features. Since SPL and F0 are both one-dimensional values, concatenating them forms a 1*2 row vector.

[0074] 3. Spectral features include formant and spectral entropy.

[0075] A. Formant Frequencies (F1, F2, F3). Formant frequencies are peaks in the spectrum, usually reflecting vowel pronunciation. They can be estimated using Linear Prediction Cepstrum Analysis (LPCC). The calculation formula is:

[0076]

[0077] Among them, F n Indicates the frequency of the nth resonance peak, n=1,2,3; LPCC n Indicates the frequency of the nth resonance peak estimated by cepstrum analysis.

[0078] B. Spectral Entropy (SE): Spectral entropy measures the degree of disorder of the spectrum and is used to describe the complexity of speech. The calculation formula is:

[0079]

[0080] Where SE represents spectral entropy; P(f) represents the probability density function corresponding to frequency f.

[0081] The calculation formula of spectral features is:

[0082]

[0083] Among them, [F1, F2, F3, SE] means the frequency of the first formant, the frequency of the second formant, the frequency of the third formant and the spectrum entropy are spliced ​​according to the time dimension; F spectral A vector representing spectral features. Since F1, F2, F3, and SE are all one-dimensional values, concatenating them forms a 1*4 row vector.

[0084] The above rhythm features, sound quality features and spectral features are integrated to form a comprehensive feature.

[0085]

[0086] Among them, Ffusion represents the comprehensive features after fusion; F prosody The vector representing the rhythmic features; F quality A vector representing the sound quality feature; F spectral Represents the vector of spectral features; w1 represents F prosody The weight of F quality The weight of F spectral When calculating the comprehensive features, it is necessary to take the vector F of the rhythmic features as prosody Add an element 0 to the end to make its dimension the same as the vector F of the spectral class feature spectral The dimension is the same; similarly, the vector F of the sound quality feature needs to be quality Add two elements 0 at the end to make its dimension the same as the vector F of the spectral class feature spectral The dimensions are the same.

[0087] Step 3: Train the Bi-LSTM-GRU model based on the extracted comprehensive features to output fatigue prediction results; the Bi-LSTM-GRU model includes an input layer, a backward layer, a forward layer, and an output layer, wherein the input layer transmits the comprehensive features of the time step to the backward layer and the forward layer respectively, the backward layer processes the reverse time series on the time step, the forward layer processes the forward time series on the time step, and the output layer splices the outputs in the forward and backward directions.

[0088] The labeled speech signals are divided into training and validation sets in a certain ratio, with a common ratio of 7:3, to ensure balanced data distribution and cover different fatigue states (fatigue and non-fatigue). The divided data has three types of features (i.e., rhythmic features, sound quality features, and spectral features). The data after feature extraction is normalized to ensure consistent feature scales, reduce the impact of numerical range differences on subsequent model training, and improve model stability and convergence speed.

[0089] Among traditional technologies, the deep learning method Bi-LSTM has achieved remarkable results in speech timing prediction tasks. It can effectively capture the temporal dependencies of speech signals and is widely used in tasks such as speech sentiment analysis and speech recognition. However, Bi-LSTM still has some limitations, mainly including the following.

[0090] (1) High computational complexity. Due to its bidirectional structure and complex gating mechanism, Bi-LSTM has high computational overhead when modeling long time series, which affects real-time performance.

[0091] (2) Gradient vanishing problem. Although LSTM has better long-term memory capability than RNN, it may still cause gradient vanishing when processing ultra-long sequence data, making it difficult to effectively transmit long-range dependency information.

[0092] (3) The model converges slowly. LSTM relies on multiple gated units to store and update states. It is prone to overfitting on small-scale datasets and takes a long time to train.

[0093] Therefore, this paper improves the Bi-LSTM structure and proposes a Bi-LSTM-GRU model. This model combines the GRU (Gated Recurrent Unit) to address the shortcomings of the LSTM. The Bi-LSTM extracts long-term temporal dependencies, maintaining its ability to model complex speech patterns. Furthermore, the GRU replaces parts of the LSTM structure, reducing computational complexity and improving training and inference efficiency. This improved Bi-LSTM-GRU model enhances the model's memory capacity, alleviates the vanishing gradient problem, and improves the model's accuracy in detecting fatigue.

[0094] like Figure 2 As shown, the Bi-LSTM-GRU model includes an input layer, a backward layer, a forward layer, and an output layer.

[0095] 1. Input layer, input comprehensive features of T time steps, comprehensive features x of each time step t They all include rhythmic features, sound quality features and spectral features.

[0096] 2. The backward layer consists of T GRU units. The comprehensive features of T time steps are input into T GRU units in a one-to-one correspondence. Each GRU unit includes an update gate and a reset gate. The calculation formula is as follows:

[0097] A. Update gate, used to determine the comprehensive feature x of the current time step input t and the hidden state h at the next time step t+1 The hidden state h at the current time step t The influence of , thereby controlling the degree of retention of previous information at the current time step,

[0098]

[0099] Among them, Z t is the activation value of the update gate; represents the sigmoid activation function; W z is the updated gate weight matrix; b z is the bias term of the update gate; h t+1 Represents the hidden state of the GRU unit from the next time step during back propagation; t∈T, T is the total number of time steps.

[0100] B. Reset gate, used to control the comprehensive feature x of the current time step input t and the hidden state h at the next time stept+1 The degree of integration,

[0101]

[0102] Among them, r t is the activation value of the reset gate; W r is to reset the gate weight matrix; b r is the bias term for the reset gate.

[0103] C. Candidate hidden state, calculate the new information of the current time step,

[0104]

[0105] in, is the candidate hidden state; tanh represents the hyperbolic tangent activation function; W h is the candidate hidden state pair x t The weight matrix of U h is the candidate hidden state pair h t+1 The weight matrix of b h is the candidate hidden state bias.

[0106] D. The final hidden state of the GRU unit,

[0107]

[0108] in, is the output (final hidden state) of the GRU unit at time step t, that is, h t .

[0109] 3. The forward layer consists of T LTSM units. The comprehensive features of T time steps are input into T LTSM units in a one-to-one correspondence. Each LSTM unit includes an input gate, a forget gate, an output gate (cell state), and a cell state. The calculation formula is as follows:

[0110] A. Input gate, used to determine the comprehensive feature x of the current time step input t and the hidden state h of the previous time step t-1 The impact on the current cell state, thereby controlling the degree of retention of subsequent information at the current time step,

[0111]

[0112] Among them, i t is the activation value of the input gate; represents the sigmoid activation function; W i is the input gate weight matrix; b irepresents the bias term of the input gate; h t-1 Represents the hidden state of the LSTM unit from the previous time step during forward propagation; t∈T.

[0113] B. Forget gate, used to control the cell state c of the previous time step t-1 What information needs to be forgotten?

[0114]

[0115] Among them, f t is the activation value of the forget gate; W f is the forget gate weight matrix; b f is the bias term of the forget gate.

[0116] C. Cell state, which determines the retention and updating of long-term memory,

[0117]

[0118]

[0119] in, is the candidate cell state at the current time step; tanh represents the hyperbolic tangent activation function; W c is the cell state weight matrix; b c is the bias term of the cell state; c t is the cell state at the current time step.

[0120] D. Output gate, used to control the hidden state h output at the current time step t , which depends on the cell state c at the current time step t ,

[0121]

[0122]

[0123] Among them, t is the activation value of the output gate; W o is the output gate weight matrix; b o is the bias term of the output gate; is the output (final hidden state) of the LTSM unit at time step t, that is, h t .

[0124] 4. The output layer consists of the output of the GRU unit and the output of the LSTM unit at each time step. The GRU unit processes the reverse time series from the Tth time step to the 1st time step, and the LSTM unit processes the forward time series from the 1st time step to the Tth time step. Finally, the outputs of the two directions are spliced ​​or summed at the same time step to obtain the bidirectional hidden state.

[0125]

[0126] Among them, H t Represents the bidirectional hidden state output at the current time step.

[0127] 5. Prediction output, after T time steps, the output layer outputs T bidirectional hidden states H t , t∈T, T bidirectional hidden states H t After the fully connected layer, it is fused into H t `, the fatigue prediction result output by the Bi-LSTM-GRU model is,

[0128]

[0129] in, is the fatigue prediction result (fatigue, non-fatigue) output by the Bi-LSTM-GRU model; Softmax represents the Softmax activation function, which converts the output into a probability form; W out is the output layer weight matrix; b out is the output layer bias term.

[0130] The threshold can be set to 0.5, when the probability value When it is greater than 0.5, the fatigue prediction result is considered to be fatigue, otherwise it is considered to be non-fatigue.

[0131] .

[0132] The Bi-LSTM-GRU model performs splicing on time steps through the back propagation of GRU units and the forward propagation of LSTM units, and finally outputs the fatigue prediction result. The GRU unit selects the weight matrix W of the candidate hidden state on multiple time steps. h , the LSTM unit selects the output gate weight matrix W at multiple time steps o However, since the GRU unit and LSTM unit are relatively reverse transmission, so at the same time step for W h and W oThe choice of is extremely important, as it affects the continuity and bidirectional compatibility of the output data in the time series. To this end, this solution designs its loss function based on the structural function of the Bi-LSTM-GRU model to constrain the accuracy of the weight selection at any time step, so that the output results can maintain continuity and compatibility in the case of bidirectional propagation fusion:

[0133]

[0134] Where L is the loss function; x t is the true comprehensive feature of the t-th time step; is the comprehensive feature predicted at the t-th time step; is a hyperparameter, , in the initial state ; For the dynamic optimization weight of the t-th time step, we have,

[0135]

[0136] in, Dynamic optimization weight for the t-1th time step; Dynamic optimization weight for the t+1th time step; Optimize weights for the initialization; is the learning rate, set ; exp is the natural exponential function; is the adjustment rate, ; t is the index of the time step.

[0137] Step 4: Collect the real-time voice data of the controller and input the real-time voice data into the trained Bi-LSTM-GRU model to predict the controller's fatigue status.

[0138] The controller's real-time voice data is collected, the corresponding comprehensive features are extracted, and the comprehensive signs are input into the trained Bi-LSTM-GRU model. The model outputs the predicted probability of fatigue status. If the probability value is greater than 0.5, the controller is judged to be "fatigued"; if the probability value is less than or equal to 0.5, the controller is judged to be "non-fatigued".

[0139] More specifically, based on the fatigue status prediction results, the system triggers the corresponding alarm mechanism. When an air traffic controller is predicted to be "fatigued," the system issues different levels of fatigue warnings based on the predicted probability value. Specifically, if the probability value is less than or equal to 0.5, the system determines that the air traffic controller is in a "non-fatigued" state and does not trigger any alarms. If the probability value is in the range of (0.5, 0.6], the system determines that the controller is in a "mild fatigue" state, "mild fatigue" being a type of "fatigue," and prompts the air traffic controller to take appropriate rest. If the probability value is in the range of (0.6, 0.7], the system determines that the controller is in a "moderate fatigue" state, "moderate fatigue" being a type of "fatigue," triggering a medium-priority fatigue warning and recommending a rotation or a short rest. If the probability value is greater than 0.7, the system determines that the controller is in a "severe fatigue" state, "severe fatigue" being a type of "fatigue," and a high-priority warning is issued, requiring the air traffic controller to immediately rest, change shifts, or perform other necessary interventions. All fatigue status predictions and alarm results are recorded in the system log to support subsequent monitoring, evaluation, and safety management analysis.

[0140] In practice, the system continuously monitors controller fatigue and adjusts fatigue prediction thresholds in real time based on feedback, ensuring accuracy and timeliness. If the system detects a change in fatigue status, relevant departments will receive an immediate alert and take necessary measures, such as shift changes and increased rest time, to protect controller health and flight safety.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A controller voice fatigue prediction method based on a remote tower scenario, characterized by: The following steps are included: Step 1: Collect audio samples containing controller voices from remote tower call data and perform denoising to obtain denoised voice signals; then label the voice signals with fatigue status. Step 2: extract prosody features, sound quality features, and spectral features from several annotated speech signals and fuse them into comprehensive features; Step 3: Training the Bi-LSTM-GRU model based on the extracted comprehensive features to output fatigue prediction results; the Bi-LSTM-GRU model includes an input layer, a backward layer, a forward layer, and an output layer, wherein the input layer transmits the comprehensive features of the time step to the backward layer and the forward layer respectively, the backward layer processes the reverse time series at the time step, the forward layer processes the forward time series at the time step, and the output layer splices the outputs in the forward and backward directions; The backward layer of the Bi-LSTM-GRU model includes T GRU units. The comprehensive features of T time steps are input into T GRU units one by one. Each GRU unit includes an update gate and a reset gate. The calculation formula is as follows: A. Update gate, used to determine the comprehensive feature x of the current time step input t and the hidden state h at the next time step t+1 The hidden state h at the current time step t The influence of , thereby controlling the degree of retention of previous information at the current time step, Among them, Z t is the activation value of the update gate; represents the sigmoid activation function; W z is the updated gate weight matrix; b z is the bias term of the update gate; h t+1 Represents the hidden state of the GRU unit from the next time step during back propagation; t∈T, T is the total number of time steps; B. Reset gate, used to control the comprehensive feature x of the current time step input t and the hidden state h at the next time step t+1 The degree of integration, Among them, r t is the activation value of the reset gate; W r is to reset the gate weight matrix; b r is the bias term for resetting the gate; C. Candidate hidden state, calculate the new information of the current time step, in, is the candidate hidden state; tanh represents the hyperbolic tangent activation function; W h is the candidate hidden state pair x t The weight matrix of U h is the candidate hidden state pair h t+1 The weight matrix of b h is the candidate hidden state bias; D. The final hidden state of the GRU unit, in, is the final hidden state output by the GRU unit at time step t; The forward layer of the Bi-LSTM-GRU model includes T LTSM units. The comprehensive features of T time steps are input into T LTSM units in a one-to-one correspondence. Each LSTM unit includes an input gate, a forget gate, an output gate, and a cell state. The calculation formula is as follows: A. Input gate, used to determine the comprehensive feature x of the current time step input t and the hidden state h of the previous time step t-1 The impact on the current cell state, thereby controlling the degree of retention of subsequent information at the current time step, Among them, i t is the activation value of the input gate; represents the sigmoid activation function; W i is the input gate weight matrix; b i represents the bias term of the input gate; h t-1 Represents the hidden state of the LSTM unit from the previous time step during forward propagation; t∈T; B. Forget gate, used to control the cell state c of the previous time step t-1 The information that needs to be forgotten, Among them, f t is the activation value of the forget gate; W f is the forget gate weight matrix; b f is the bias term of the forget gate; C. Cell state, which determines the retention and updating of long-term memory, in, is the candidate cell state at the current time step; tanh represents the hyperbolic tangent activation function; W c is the cell state weight matrix; b c is the bias term of the cell state; c t is the cell state at the current time step; D. Output gate, used to control the hidden state h output at the current time step t , which depends on the cell state c at the current time step t , Among them, t is the activation value of the output gate; W o is the output gate weight matrix; b o is the bias term of the output gate; is the final hidden state output by the LTSM unit at time step t; The output layer of the Bi-LSTM-GRU model consists of the output of the GRU unit and the output of the LSTM unit at each time step, where the GRU unit processes the reverse time series from the Tth time step to the 1st time step, and the LSTM unit processes the forward time series from the 1st time step to the Tth time step; finally, the outputs of the two directions are spliced ​​or summed at the same time step to obtain the bidirectional hidden state. Among them, H t Represents the bidirectional hidden state output at the current time step; After T time steps, the output layer outputs T bidirectional hidden states H t , t∈T, T bidirectional hidden states H t After the fully connected layer, it is fused into H t `, the fatigue prediction result output by the Bi-LSTM-GRU model is, in, is the fatigue prediction result output by the Bi-LSTM-GRU model; Softmax represents the Softmax activation function; W out is the output layer weight matrix; b out is the output layer bias term; Step 4: Collect the real-time voice data of the controller and input it into the trained Bi-LSTM-GRU model to predict the controller's fatigue status; The loss function of the Bi-LSTM-GRU model is: Where L is the loss function; x t is the true comprehensive feature of the t-th time step; is the comprehensive feature predicted at the t-th time step; is a hyperparameter, , in the initial state ; For the dynamic optimization weight of the t-th time step, we have, in, Dynamic optimization weight for the t-1th time step; Dynamic optimization weight for the t+1th time step; Optimize weights for the initialization; is the learning rate, set ; exp is the natural exponential function; is the adjustment rate, ; t is the index of the time step.

2. The method for predicting controller voice fatigue based on remote tower scenario according to claim 1 is characterized in that: In step 1, the step of collecting audio samples containing controller voices from remote tower call data and performing denoising processing to obtain denoised voice signals includes: The time-frequency domain representation of the original speech signal in the collected audio samples is: Where f is frequency; t is time; x(t) is the original speech signal; X(f,t) is the complex value of the original speech signal at time t and frequency f; w(t-t0) is the window function; t0 is the center time of the current window; F is the Fourier transform operation; Assume that the noise spectrum is estimated to be , denoising is performed by spectral subtraction to obtain the denoised speech signal, Where |X(f,t)| 2 is the power spectrum of the original speech signal; is the power spectrum of the noise estimate; S(f,t) is the power spectrum of the speech signal after denoising; Perform inverse short-time Fourier transform on the power spectrum S(f,t) of the denoised speech signal to obtain the corresponding time domain signal. in, represents the time domain signal corresponding to S(f,t); F -1 Represents an inverse short-time Fourier transform operation.

3. The method for predicting controller voice fatigue based on remote tower scenario according to claim 1 is characterized in that: In step 1, the step of marking the fatigue state of the speech signal includes: Conduct subjective and objective fatigue tests on the controllers' voice signals. The subjective fatigue assessment is based on the Samn-Perelli 7-level fatigue scale and the SART scale; the objective fatigue assessment is based on the psychomotor alertness test. The SP score of the controller was evaluated based on the Samn-Perelli 7-level fatigue scale. score ; SART score for evaluating controllers based on the SART scale score ; Scored RT of controllers assessed based on the psychomotor vigilance test score ; Calculate the comprehensive fatigue score, Among them, Fatigue score Indicates the comprehensive fatigue score; when the comprehensive fatigue score Fatigue score When the set threshold is exceeded, the subject is marked as being in a fatigue state.

4. The method for predicting controller voice fatigue based on remote tower scenario according to claim 1, characterized in that: In step 2, the step of extracting prosodic features from a number of annotated speech signals includes: Prosodic features include speech rate, pause frequency, and pause duration; The formula for calculating speaking speed is: Among them, SR represents the speaking speed; N s Indicates the total number of syllables pronounced in the speech sample; T d Indicates the duration of the speech sample segment; The calculation formula for the pause frequency is: Among them, PF represents the pause frequency; N p represents the number of pauses in the speech sample; T d Indicates the duration of the speech sample segment; The calculation formula for the pause duration is: Among them, PD represents the pause duration; Indicates the duration of each pause; i represents the index of the i-th pause, i=1,2,...,N p ; The calculation formula of rhythmic features is: Among them, [SR, PF, PD] means splicing the speech rate, pause frequency and pause duration according to the time dimension; F prosody A vector representing prosodic features.

5. The method for predicting controller voice fatigue based on remote tower scenario according to claim 4 is characterized in that: In step 2, the step of extracting sound quality features from a number of annotated speech signals includes: Sound quality characteristics include sound pressure level and fundamental frequency; The formula for calculating the sound pressure level is, Where SPL represents the sound pressure level; p represents the current sound pressure; p0 represents the reference sound pressure; The fundamental frequency is calculated as, Among them, F0 represents the basic frequency; N cycles Indicates the number of vibrations in a cycle; N period Indicates the cycle duration; The calculation formula for sound quality features is: Among them, [SPL,F0] represents the concatenation of sound pressure level and fundamental frequency in the time dimension; F quality A vector representing the sound quality features.

6. The method for predicting controller voice fatigue based on remote tower scenario according to claim 5 is characterized in that: In step 2, the step of extracting spectral features from a number of annotated speech signals includes: Spectral features include formant and spectral entropy; The calculation formula of the resonance peak is, Among them, F n Indicates the frequency of the nth resonance peak, n=1,2,3; LPCC n Indicates the frequency of the nth resonance peak estimated by cepstrum analysis; The calculation formula of spectral entropy is: Where SE represents spectral entropy; P(f) represents the probability density function corresponding to frequency f; The calculation formula of spectral features is: Among them, [F1, F2, F3, SE] means the frequency of the first formant, the frequency of the second formant, the frequency of the third formant and the spectrum entropy are spliced ​​according to the time dimension; F spectral A vector representing spectral features.

7. The method for predicting controller voice fatigue based on remote tower scenario according to claim 6, characterized in that: In step 2, the step of fusing into comprehensive features includes: The rhythm features, sound quality features and spectral features are integrated to form comprehensive features. Among them, F fusion Represents the comprehensive features after fusion; w1 represents F prosody The weight of F quality The weight of F spectral The weight of .

Images