Pilot earphone hearing protection method and system based on voice recognition compensation

Abstract

The invention relates to the field of aviation voice signal processing, and discloses a voice recognition compensation pilot earphone hearing protection method and system, and the method comprises the following steps: collecting multi-modal data, and separating a sound source through tensor decomposition; inferring a pilot state by using a dynamic network; performing context recognition and semantic evaluation on the attention target voice; and finally, dynamically modulating the sound field based on deep reinforcement learning, enhancing the voice in a personalized manner, and outputting after noise suppression. The system comprises a multi-mode perception data acquisition module, a sound source decoupling module, a pilot state inference module, a voice processing and semantic evaluation module and a sound field modulation and output module. According to the invention, high-fidelity speech extraction is realized through multi-modal perception and tensor decomposition; evaluating priority key information in combination with attention and semantics; and deep reinforcement learning and model prediction control are adopted to dynamically optimize the sound field, so that the voice recognition accuracy and the pilot information acquisition efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of aviation voice signal processing, specifically to a method and system for hearing protection of pilot headsets with voice recognition compensation. Background Technology

[0002] The inherently high-intensity and complex noise environment of the pilot's cockpit poses a continuous challenge to the clarity of voice communication and the pilot's hearing health. Although traditional passive sound insulation and active noise cancellation technologies are used, they are still insufficient in dealing with low-frequency strong noise, non-stationary noise, and personalized auditory optimization, making it difficult to completely guarantee the pure transmission of the target speech.

[0003] Existing speech enhancement algorithms often suffer from speech distortion or artifacts when processing cockpit signals with extremely low signal-to-noise ratios, affecting the naturalness and intelligibility of conversations. Furthermore, these technologies typically lack the ability to deeply analyze complex acoustic environments, making it difficult to effectively distinguish and extract multiple mixed target sound sources from interfering sources. When automatic speech recognition technology is directly applied to the cockpit, its accuracy and robustness are severely limited. Aviation-specific terminology, pilot accents, and strong noise interference all contribute to the decline in recognition performance. More importantly, current systems generally lack awareness and utilization of flight context and the pilot's real-time status, failing to intelligently assist in information acquisition.

[0004] Pilots, under high cognitive load, still need to sift through a large amount of auditory information to identify key instructions, which not only increases their workload but also carries the risk of missing important information. Current technology has failed to provide an effective mechanism to dynamically optimize auditory presentation based on the importance of information and the pilot's attention level, nor has it adequately considered personalized hearing protection needs.

[0005] Therefore, this invention proposes a method and system for hearing protection of pilot headsets with voice recognition compensation to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and system for hearing protection of pilot headsets with voice recognition compensation. It solves the problem of how pilots can achieve clear and efficient voice communication, effective hearing protection, and reduced cognitive load in complex and high-noise cockpit environments through intelligent sensing, precise sound source processing, and personalized sound field modulation.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for hearing protection of pilot headsets with voice recognition compensation, the method comprising the following steps:

[0008] S1. Collect multi-channel acoustic signals of the pilot's surrounding environment and simultaneously acquire flight parameters to form multimodal sensing data;

[0009] S2. Apply a sound source decoupling algorithm based on tensor decomposition to analyze the acoustic scene of the multi-channel acoustic signals in the multimodal sensing data, and separate the target speech sound source and the environmental noise sound source.

[0010] S3. Input the flight parameters, the acoustic characteristics of the target speech source and the environmental noise source in the multimodal perception data into the pre-trained dynamic attention network model to infer the pilot's current attention flow and cognitive state.

[0011] S4. For the target speech source whose pilot attention flow is determined, combine the flight parameters in the multimodal perception data as context to perform context-aware automatic speech recognition to obtain the recognition text, and evaluate the semantic importance of the recognition text based on the aviation domain knowledge graph to obtain the semantic importance result.

[0012] S5. Based on the pilot's attention flow, cognitive state, and semantic importance, a sound field modulation strategy based on deep reinforcement learning is applied to personalize the target speech source and adaptively suppress the environmental noise source. The processed sound source is then synthesized into a binaural audio signal and output to the pilot's headphones.

[0013] Preferably, in step S1, the step of collecting multi-channel acoustic signals of the pilot's surrounding environment and simultaneously acquiring flight parameters to form multimodal perception data also includes collecting the pilot's head posture data or physiological state data.

[0014] Preferably, in step S2, the step of applying a tensor decomposition-based sound source decoupling algorithm to analyze the acoustic scene of the multi-channel acoustic signal includes:

[0015] The multi-channel acoustic signals are transformed from the time domain to the time-frequency domain and constructed into a time-frequency domain observation signal tensor. In the formula, K represents the number of time frames, F represents the number of frequency points, and M represents the number of frequency points. total Indicates the total number of microphone channels;

[0016] The observed signal tensor was analyzed using a normalized multiple regression model. The sound sources are decomposed to separate them; this decomposition is expressed by the CPD model formula:

[0017]

[0018] In the formula, N src λ represents the estimated number of independent sound sources; r is the index of the sound source; λ r Represents the complex amplitude of the r-th sound source; This represents the column vector of time activation modes for the r-th sound source. This represents the frequency response column vector of the r-th sound source; Represents the column vector of microphones or spatial factors for the r-th sound source; ° represents the outer product operation of tensors;

[0019] The factors of each sound source obtained based on the CPD model decomposition, namely and The target speech source and the environmental noise source are reconstructed separately.

[0020] Preferably, in step S3, the pre-trained dynamic attention network model includes a graph neural network layer and a recurrent neural network layer with subsequent connections, used to process the acoustic characteristics of the flight parameters, the target speech source, and the environmental noise source. The pre-trained dynamic attention network model further infers the pilot's current attention flow and cognitive state based on the pilot's head posture data or physiological state data.

[0021] The GNN layer constructs graph-structured data from the flight parameters, the acoustic characteristics of the target speech source, and the environmental noise source. Different information entities serve as nodes in the graph, and a graph attention mechanism is applied to learn the relationships between the nodes to update the representation of each node. After the update of any target node v in time frame n, the node representation h′ is obtained. v,n Calculation using the formula updated via GNN nodes:

[0022]

[0023] In the formula, h′ v,n Let σ be the node representation of target node v after the update in time frame n; GNN The non-linear activation function of the GNN layer is used; the summation iterates through all source nodes u, where the source nodes u are selected from the set of neighbor nodes of the target node v. Or the target node v itself; α vu,n W represents the attention coefficient of the source node u to the target node v at time frame n, calculated using the attention mechanism. GNN h is the learnable weight matrix of the GNN layer; u,n This is the input feature vector of the source node u in time frame n;

[0024] The RNN layer receives the updated node representation output by the GNN layer and combines it with temporal information to capture dynamic change patterns through the transmission of internal network states, ultimately outputting results that characterize the pilot's attention flow and cognitive state.

[0025] Preferably, in step S4, when processing the target speech source determined by the pilot's attention flow, the target speech source, together with the flight parameters in the multimodal perception data, is used as context information and input into a context-aware automatic speech recognition model to perform speech recognition operations and generate corresponding recognized text.

[0026] The generated identified text is matched and semantically analyzed with a pre-configured aviation domain knowledge graph, which stores aviation-specific vocabulary, standard instruction structures, or identifiers representing urgency. This analysis process is used to evaluate the semantic importance of the identified text and ultimately obtain a quantified semantic importance result.

[0027] Preferably, in step S5, the sound field modulation strategy based on deep reinforcement learning is applied, specifically referring to:

[0028] A deep reinforcement learning agent is used, which takes the pilot's attention flow, cognitive state and semantic importance as part of the input state for its decision-making, and optimizes the action strategy by interacting with the acoustic environment, aiming to maximize a predefined cumulative reward.

[0029] The instantaneous reward R obtained by the DRL agent at each decision time n n Calculated using the instantaneous reward function formula:

[0030] R n =β clr ·M clarity (n)+β prot ·M protection (n)-β load ·P overload (L cog (n))+β comm ·M comm_eff (n)-β ctrl ·P action_cost (A n );

[0031] In the formula, R n M is the instantaneous reward value evaluated at decision time n; clarity (n) represents the quantitative indicator of the target speech clarity at decision time n; M protection (n) represents a quantitative indicator of the effectiveness of hearing protection for the pilot at decision time n; P overload (L cog (n) represents a value based on the current pilot's cognitive load L. cog (n) The overload penalty function is calculated; M comm_eff (n) represents a quantitative indicator of communication efficiency at decision time n; Paction_cost (A n ) represents the sound field modulation action A selected by the current DRL agent. n The penalty function imposed on cost or smoothness; L cog (n) represents the pilot's cognitive state; A n It is the sound field modulation action output by the DRL agent at time n; while β clr ,β prot ,β load ,β comm ,β ctrl These are the preset weighting coefficients used to balance the importance of the above indicators;

[0032] The DRL agent makes decisions based on the learned action strategy and outputs a set of specific sound field modulation parameters. These parameters are used to guide the personalized enhancement processing of the target speech source and the adaptive suppression processing of the environmental noise source.

[0033] Preferably, in step S5, the adaptive suppression of the environmental noise source is achieved through a model predictive control framework;

[0034] The MPC framework utilizes short-term prediction information obtained by analyzing the characteristics of the isolated environmental noise sources, and combines it with the inferred pilot attention flow to optimize the generation of anti-phase signals in the active noise cancellation system.

[0035] The goal of the MPC framework is to minimize a cost function within a preset prediction time domain. The model predicts the control cost function J. MPC Represented as:

[0036]

[0037] In the formula, J MPC t0 represents the cost function value of model predictive control; t0 represents the current discrete time point of MPC control; i represents the step size index in the prediction time domain; P mpc It is the length of the prediction time domain; y res (t0+i|t0) represents the state vector of residual noise at the pilot's headset at the i-th future step predicted from the current time t0; y ref (t0+i|t0) represents the expected residual noise target state vector at the i-th future step, set at the current time t0; u anr (t0+i) represents the anti-phase signal control vector decided at the current time t0 and planned to be applied in the i-th step in the future; Q mpc R represents the weight matrix used to penalize residual noise error. mpc This represents the weight matrix used to penalize the control input energy.

[0038] The present invention also provides a voice recognition compensation-based hearing protection system for pilot headsets, the system comprising the following modules:

[0039] The multimodal perception data acquisition module is configured to collect multi-channel acoustic signals from the pilot's surrounding environment and simultaneously acquire flight parameters to form multimodal perception data.

[0040] The sound source decoupling module is configured to use a tensor decomposition-based sound source decoupling algorithm to analyze the acoustic scene of the multi-channel acoustic signals in the multimodal sensing data, thereby separating the target speech sound source and the environmental noise sound source.

[0041] The pilot state inference module is configured to input the flight parameters, the acoustic characteristics of the target speech source and the environmental noise source in the multimodal perception data into a pre-trained dynamic attention network model to infer the pilot's current attention flow and cognitive state.

[0042] The speech processing and semantic evaluation module is configured to determine the target speech source for the pilot's attention flow, combine the flight parameters in the multimodal perception data as context to perform context-aware automatic speech recognition to obtain the recognized text, and evaluate the semantic importance of the recognized text based on the aviation domain knowledge graph to obtain the semantic importance result.

[0043] The sound field modulation and output module is configured to, based on the pilot's attention flow, cognitive state, and semantic importance results, use a sound field modulation strategy based on deep reinforcement learning to personalize the target speech source and adaptively suppress the environmental noise source. Then, the processed target speech source and environmental noise source are synthesized into a binaural audio signal and output to the pilot's headphones.

[0044] This invention provides a method and system for hearing protection of pilot headsets with voice recognition compensation. It has the following beneficial effects:

[0045] 1. This invention utilizes multimodal sensing that fuses multi-channel acoustic signals with flight parameters, combined with a precise sound source decoupling technique based on tensor decomposition, to achieve high-fidelity extraction and clear presentation of target speech in extremely noisy and interwoven sound environments such as flight cockpits. Compared to existing technologies that rely on single microphone input or traditional spectral subtraction and other noise suppression methods, which are prone to speech signal submersion and severe target speech distortion under strong interference, this invention solves the problem of failing to ensure clear transmission of critical flight commands.

[0046] 2. This invention uses a dynamic attention network model to infer the pilot's attention flow and cognitive state, and combines this with an innovative design that uses aviation knowledge graphs to assess the semantic importance of identified text. This allows for the identification and priority presentation of voice information crucial to the flight mission, assisting pilots in quickly focusing when overwhelmed by information. Compared to traditional voice systems that treat all voice information equally or perform only simple keyword detection, which fails to effectively differentiate information priorities, this invention overcomes the shortcomings of traditional methods that may lead to excessive cognitive load for pilots or even missed critical commands.

[0047] 3. This invention employs a sound field modulation strategy based on deep reinforcement learning, combined with a unique method of adaptive suppression of environmental noise through model predictive control, creating a personalized and dynamically optimized auditory space for pilots. It enhances the target speech while effectively reducing background noise interference, and simultaneously protects hearing. Compared to existing headphones that use fixed noise reduction parameters or simple adaptive filtering, lacking a sophisticated understanding of individual states and the environment, this invention solves the problem of their inability to dynamically adjust the acoustic presentation based on the pilot's real-time state and semantic information, resulting in poor auditory experience or insufficient hearing protection.

[0048] 4. This invention integrates flight parameters as contextual information into the automatic speech recognition process and utilizes aviation knowledge graphs to assist semantic understanding, thereby improving the accuracy and reliability of speech recognition in aviation-specific domains and high-noise environments. Existing general-purpose automatic speech recognition systems, when directly applied to the cockpit, often perform poorly due to a lack of domain knowledge and contextual awareness. This invention effectively improves the situation where frequent recognition errors occur in complex flight environments, making it difficult to meet the high reliability requirements of aviation communications.

[0049] 5. This invention systematically integrates multiple technical aspects such as multimodal perception, intelligent sound source analysis, pilot state understanding, and dynamic sound field optimization, forming a closed-loop intelligent hearing assistance system. This system comprehensively improves the pilot's information acquisition efficiency and decision-making ability in complex mission environments. Traditional solutions often focus only on single aspects such as speech enhancement or noise reduction, lacking systematic integration and intelligent collaboration. This invention overcomes the shortcomings of isolated functional modules that cannot work together to cope with dynamically changing flight environments and pilot needs, thereby indirectly improving flight safety margins. Attached Figure Description

[0050] Figure 1 This is a flowchart of the method of the present invention;

[0051] Figure 2 This is a system architecture diagram of the present invention. Detailed Implementation

[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Please see Figure 1 This invention provides a method for hearing protection of pilot headsets with voice recognition compensation, the method comprising the following steps:

[0054] S1. Collect multi-channel acoustic signals of the pilot's surrounding environment and simultaneously acquire flight parameters to form multimodal sensing data;

[0055] In this embodiment, step S1 aims to comprehensively and synchronously collect multi-dimensional information related to the pilot and the cockpit environment. This information together constitutes the original data basis for subsequent acoustic scene analysis, pilot state inference, and personalized sound field modulation.

[0056] Specifically, in the acquisition of the multimodal sensing data, the system captures multi-channel acoustic signals in real time through multiple microphone channels deployed in the acoustic environment surrounding the pilot. These microphone channels can preferably be integrated inside or outside the pilot's helmet system, or distributed in key locations within the cockpit to form a microphone array. The configuration of this microphone array is designed to receive sound waves from different spatial angles, thereby enabling the recording of various environmental noises, including the pilot's own voice, air traffic controller instructions, conversations between other crew members in the cockpit, and sounds such as engine roar, airflow noise, and onboard equipment operation. The acquired multi-channel acoustic signals not only contain sound intensity information, but more importantly, they carry spatial location clues of the sound sources, such as the time difference or intensity difference between different microphones receiving signals from the same sound source. These spatial clues are crucial for accurately separating different sound sources in subsequent steps.

[0057] Simultaneously with the acquisition of the multi-channel acoustic signals, flight parameters are acquired. These flight parameters are obtained through data exchange with the data bus interface of the airborne avionics system, or measured by dedicated sensors. The flight parameters typically include, but are not limited to:

[0058] The aircraft's current flight altitude, indicated airspeed or ground speed, attitude information such as pitch angle, roll angle and yaw angle, engine speed or thrust percentage, autopilot operating mode, and status indications of the warning system.

[0059] These flight parameters reflect the aircraft's current operational status, flight phase, and dynamic changes in the external environment in real time, providing crucial contextual information for understanding the background of acoustic events, assessing the urgency of the speech content, and inferring the pilot's cognitive load.

[0060] The acquired multi-channel acoustic signals and flight parameters are all precisely timestamped to ensure that the data are strictly synchronized and aligned in time series, thus merging into a unified multimodal sensing data stream.

[0061] To further enhance the depth and accuracy of pilot state perception, this step may further include collecting pilot head posture data or physiological state data.

[0062] Data on the pilot's head posture can be collected, for example, through an inertial measurement unit (IMU) or a miniature camera integrated into the pilot's helmet. The IMU can output the pilot's head orientation angles (such as pitch, yaw, and roll angles) and angular velocity information in real time. This data is directly related to the area where the pilot's visual attention may be focused, thereby helping to determine the direction of the sound source or visual target that the pilot is paying attention to.

[0063] Collecting physiological data on pilots can be achieved, for example, through biosensors worn on specific parts of the pilot's body or electrodes integrated into the helmet liner. These sensors can monitor the pilot's heart rate, heart rate variability, skin conductivity, body temperature, or, in some implementations, specific frequency bands of brain electrical activity. These physiological indicators can indirectly reflect the pilot's physiological arousal level, stress level, fatigue state, and cognitive load, providing an objective basis for assessing the pilot's overall condition.

[0064] Both head posture data and physiological state data require strict time synchronization with acoustic signals and flight parameters during acquisition, and must be integrated into the multimodal perception data to provide more direct and richer input features for accurate inference of the pilot's attention flow and cognitive state. By integrating this multimodal data, this invention can more comprehensively characterize the pilot and the complex environment they are in, laying a solid data foundation for subsequent intelligent compensation and protection strategies.

[0065] S2. Apply a sound source decoupling algorithm based on tensor decomposition to analyze the acoustic scene of the multi-channel acoustic signals in the multimodal sensing data, and separate the target speech sound source and the environmental noise sound source.

[0066] In this embodiment, the core task of step S2 is to perform refined analysis on the multi-channel acoustic signals acquired in step S1, in order to separate the target speech source carrying key information and the environmental noise source that interferes with communication and hearing. This step employs a source decoupling algorithm based on tensor decomposition to address the complex and dynamically changing acoustic environment inside the cockpit.

[0067] In one possible implementation, the sound source decoupling process specifically includes:

[0068] The multi-channel acoustic signals acquired from multimodal sensing data are converted from a time-domain representation to a time-frequency domain representation. This is typically achieved by applying a short-time Fourier transform (STFT) to the signal of each microphone channel, which involves dividing a long-duration signal into several shorter time frames and performing a Fourier transform on each frame to obtain the joint distribution of the signal in time and frequency.

[0069] After time-frequency conversion, the complex-valued responses of all microphone channels at various time frames and frequency points are organized and constructed into a third-order time-frequency domain observation signal tensor, denoted as . Should In the formula, K represents the total number of time frames selected for analysis, F represents the total number of discrete frequency points, and M... total This represents the total number of microphone channels used to acquire acoustic signals. This tensor... It fully characterizes the observed multi-channel acoustic scene.

[0070] Subsequently, the constructed time-frequency domain observed signal tensor χ was decomposed using a canonical-polyyadic-decomposition (CPD) model, also known as parallel factor analysis (PARAFAC). The basic idea of ​​the CPD model is to approximate a high-order tensor as the sum of several rank tensors, each rank tensor corresponding to an independent potential sound source component. Its mathematical decomposition form can be expressed by the following CPD model formula:

[0071]

[0072] In the formula, N src The number of independent sound sources is estimated or predetermined, which can be determined based on prior knowledge or through model selection criteria; r is the index of the sound source, ranging from 1 to N. src ;λ r The complex amplitude or weight of the r-th sound source component reflects the strength of the contribution of that sound source to the overall signal. This represents the time activation mode column vector of the r-th sound source. Each element of this vector describes the activity level or energy envelope of the sound source in the corresponding time frame. The frequency response column vector represents the r-th sound source, and its elements characterize the energy distribution or spectral shape of the sound source at different frequency points. This represents the column vector of microphones or spatial factors for the r-th sound source. This vector captures the spatial characteristics of the sound source relative to the microphone array, such as the relative gain and phase difference of each channel. This information is closely related to the direction of arrival (DOA) of the sound source; (symbol) The outer product operation represents tensors. This is achieved by minimizing the observed tensor. A measure of error between the CPD model approximation and the model (such as the Frobenius norm) can be used to iteratively solve for the factor vectors of each sound source. and and amplitude λ r .

[0073] Once the CPD model decomposition is complete, the feature factors corresponding to each independent sound source r are obtained. and Then, these factors can be used to reconstruct the signals of each separated sound source. For example, the representation of the r-th sound source in the time-frequency domain can be obtained by multiplying the outer product of its three factor vectors by its amplitude λ. r This is approximated. Subsequently, by performing an inverse short-time Fourier transform (ISTFT) on these reconstructed time-frequency domain sound source signals, the individual sound source signal in the time domain can be obtained.

[0074] Based on the characteristics of these isolated sound source signals (e.g., their spectral features, statistical characteristics of time activation patterns, or spatial location information inferred from spatial factors), combined with possible prior knowledge (such as the spectral range of typical speech or the general direction of a pilot's speech), they can be distinguished into target speech sources (such as pilot's conversations or tower instructions) and environmental noise sources (such as engine noise, wind noise, etc.).

[0075] In some embodiments, to improve the robustness and accuracy of sound source decoupling, additional constraints or regularization terms can be introduced into the tensor decomposition algorithm, such as based on prior knowledge of the sparsity, smoothness, or nonnegativity of the sound source.

[0076] Furthermore, the pilot's attention flow information inferred from subsequent step S3, or the semantic importance and recognition confidence of specific speech signals evaluated from subsequent step S4, can also serve as feedback to guide the sound source decoupling process in step S2. For example, if the pilot is detected to be highly focused on a specific direction, the parameters of the CPD model or the subsequent reconstruction process can be adjusted to prioritize or more precisely separate sound sources from that direction. If the recognition confidence of a certain important speech signal is low, iterative enhancement separation of that sound source can be triggered, for example, by adjusting N... src The estimation or imposing stronger constraints on specific factors.

[0077] S3. Input the flight parameters, the acoustic characteristics of the target speech source and the environmental noise source in the multimodal perception data into the pre-trained dynamic attention network model to infer the pilot's current attention flow and cognitive state.

[0078] In this embodiment, step S3 aims to extract key information about the pilot's internal state from multimodal perception data. Specifically, it dynamically infers the pilot's current attention flow and cognitive state. This process is achieved by inputting the various types of information obtained in the preceding steps into a pre-trained dynamic attention network model.

[0079] Generally, the input data of the pre-trained dynamic attention network model includes: flight parameters from the multimodal perception data in step S1, such as the current flight altitude, speed, attitude, etc.; and acoustic characteristics of the target speech source and the environmental noise source separated in step S2, such as the energy envelope, spectral features, estimated spatial location or direction information of these sources.

[0080] In one possible implementation, if additional head posture data (such as head orientation angle) or physiological state data (such as heart rate, skin conductance response, etc.) of the pilot are collected in step S1, these data that directly reflect the pilot's behavior or physiological response will also be used as important feature inputs and fed into the dynamic attention network model to enhance the accuracy and reliability of attention flow and cognitive state inference.

[0081] Specifically, the pre-trained dynamic attention network model preferably includes a graph neural network (GNN) layer and a subsequent recurrent neural network (RNN) layer to jointly process and understand these heterogeneous multimodal input data.

[0082] The main function of the GNN layer is to perform structured representation learning and information fusion on various information entities, such as input flight parameters, acoustic characteristics, and possible head posture or physiological data. In this embodiment, different information entities (e.g., a specific flight parameter such as "altitude," the "energy characteristics" of a separated sound source, or the pilot's "head yaw angle") are constructed as nodes in a graph structure data. The edges between these nodes can be defined based on their inherent or learned correlations, such as temporal synchronization, potential causal relationships, or semantic relevance. The GNN layer learns the degree of mutual influence between different nodes by applying a graph attention mechanism and aggregating information from neighboring nodes to update the representation of each node itself. The updated node representation h′ of any target node v at a specific time frame n is... v,n The following GNN node update formula can be used for calculation:

[0083]

[0084] In the formula, h′ v,n σ is the feature representation vector updated by the target node v after processing by the GNN layer at time frame n; GNN This represents the non-linear activation function used in the GNN layer, such as the Corrected Linear Unit (ReLU) or the Gaussian Error Linear Unit (GELU); the summation operation iterates through all relevant source nodes u, which are selected from the set of neighbor nodes of the target node v. Or the target node v itself (i.e., containing a self-loop); α vu,n W represents the attention coefficient of the source node u for the target node v at time frame n, calculated using an attention mechanism. This coefficient reflects the importance of the information of the source node u for updating the representation of the target node v in the current context. GNN The learnable weight matrix for the GNN layer is optimized during model training; h u,n Let be the input feature vector of the source node u at time frame n. In this way, GNN can effectively integrate information from different modalities and capture the complex dependencies between them.

[0085] The RNN layer, which can be, for example, a Long Short-Term Memory (LSTM) network unit or a Gated Recurrent Unit (GRU), is responsible for processing the sequence of node representations updated at each time frame, output by the GNN layer. Through its internal recursive state propagation mechanism, the RNN layer can effectively capture the dynamic evolution patterns and long-term dependencies of these feature representations over time. It temporally integrates the snapshots of the GNN output at each time point, which incorporate multimodal information, thereby understanding the continuous changes in the pilot's state over time.

[0086] The output of the RNN layer is a comprehensive inference of the pilot's current attention flow (e.g., determining whether the pilot is focused on listening to tower instructions, intercom, or alert to an alarm source in a specific direction) and cognitive state (e.g., assessing whether the pilot's cognitive load is too high, moderate, or low, and whether there are signs of distraction or fatigue). This dynamic attention network model learns the mapping relationship from input multimodal features to pilot state by pre-training on a dataset containing a large number of labeled samples.

[0087] S4. For the target speech source whose pilot attention flow is determined, combine the flight parameters in the multimodal perception data as context to perform context-aware automatic speech recognition to obtain the recognition text, and evaluate the semantic importance of the recognition text based on the aviation domain knowledge graph to obtain the semantic importance result.

[0088] In this embodiment, step S4 focuses on in-depth processing of the target speech source currently of interest to the pilot, including converting its content into text and further evaluating the semantic importance of this text information in a specific aviation context. This step is of great significance for subsequent sound field modulation and possible information highlighting or decision support.

[0089] Specifically, when step S3 infers the current direction of the pilot's attention and thereby identifies one or more target speech sources (e.g., the pilot is focused on receiving instructions from air traffic controllers or listening to critical announcements from other crew members in the cockpit), the system will select these attention-indicated target speech sources (which have been effectively separated in step S2) as processing objects.

[0090] In one possible implementation, for each identified target speech source, the system feeds its acoustic signal, along with relevant flight parameters from the multimodal sensing data acquired in step S1, as input to a context-aware Automatic Speech Recognition (ASR) model. These flight parameters, such as the aircraft's current flight phase (e.g., takeoff, cruise, approach, landing), geographical location, altitude, speed, and current weather conditions, provide rich contextual information for the ASR model. This context-aware capability allows the ASR model to dynamically adjust its acoustic or language model to better adapt to the specific aviation communication environment. For example, it can prioritize recognizing terms and instructions relevant to a specific flight phase, or adjust its adaptation to accents or speech rates based on known communication parties (e.g., specific control towers). This improves the accuracy and robustness of speech recognition in complex cockpit noise environments and in the face of possible variations in accents and speech rates, and generates corresponding recognized text.

[0091] After obtaining the recognized text output by the context-aware automatic speech recognition model, the system performs semantic analysis and importance assessment on the text. This assessment process relies on a pre-configured aviation domain knowledge graph. This aviation domain knowledge graph is a structured knowledge base that systematically stores a large amount of professional knowledge and information entities related to aviation operations and their interrelationships. This information may include, but is not limited to:

[0092] Aviation-specific vocabulary and abbreviations: for example, standard aviation terminology, verbs and nouns related to flight operations, commonly used abbreviations and their full names, etc.

[0093] Standard command structure and communication paradigm: For example, the typical syntax structure of air traffic control commands, the template of standard communication procedures between pilots and controllers, and the command format for specific operations (such as altitude changes and heading adjustments).

[0094] Identifying information that indicates urgency or safety criticality: for example, clear emergency call signs (such as “MAYDAY”, “PAN-PAN”), warning words directly related to flight safety (such as “stall”, “collision”, “fire”), and phrases indicating potential risks or abnormal situations.

[0095] Flight phase and mission-related information: For example, common or expected communication content and instruction types in different flight phases (such as pre-flight checks, climb, cruise, descent, and post-landing).

[0096] By performing complex matching, pattern recognition, entity linking, and semantic reasoning on the generated identified text against entries in this aviation-related knowledge graph, the system can analyze the semantic content of the identified text and assess its importance in the current flight context. For example, if the identified text contains high-priority emergency keywords, or its structure perfectly matches the format of a key flight instruction, its semantic importance will be rated as high. Conversely, if the identified content is non-mission-related casual conversation or routine background confirmation, its semantic importance may be low.

[0097] This analysis process ultimately outputs a quantified semantic importance result. This result can be a numerical score, a predefined level (such as "high," "medium," or "low" importance), or a multidimensional vector containing semantic category and confidence level. This semantic importance result will serve as one of the important bases for sound field modulation decisions in the subsequent step S5, enabling the system to dynamically adjust the enhancement level of the target speech according to the importance of the information.

[0098] In some embodiments, if the semantic importance assessment result of the identified text is high, but the confidence level of the identified text given by the ASR model is low, this information can serve as a feedback signal, indicating that more refined iterative separation processing may be required for the target speech source in step S2, or the decoding strategy of the ASR model may be adjusted in step S4 to try to improve the recognition accuracy.

[0099] S5. Based on the pilot's attention flow, cognitive state and semantic importance results, a sound field modulation strategy based on deep reinforcement learning is applied to personalize the target speech source and adaptively suppress the environmental noise source. Then, the processed sound source is synthesized into a binaural audio signal and output to the pilot's headphones.

[0100] In this embodiment, step S5 is the core decision-making and execution link of the entire method. Its goal is to dynamically and intelligently adjust the pilot's auditory environment based on the pilot's attention flow, cognitive state, and semantic importance of the target speech obtained in the aforementioned steps. Specifically, it applies a sound field modulation strategy based on deep reinforcement learning to personalize the target speech source of interest and adaptively suppress environmental noise sources. Finally, the optimized sound source is synthesized into a binaural audio signal and output to the pilot's headphones, in order to ensure communication clarity while taking into account hearing protection and cognitive load management.

[0101] Specifically, the application of a sound field modulation strategy based on Deep Reinforcement Learning (DRL) refers to using a DRL agent to learn and execute optimal sound field adjustment actions. This DRL agent uses the pilot's attention flow and cognitive state inferred in step S3, and the semantic importance results evaluated in step S4, as part of its input state for decision-making. The agent continuously learns through interaction with a simulated or calibrated real acoustic environment, exploring different combinations of sound field modulation parameters and optimizing its action policy according to a preset reward mechanism, aiming to maximize a long-term cumulative expected reward.

[0102] The instantaneous reward R obtained by the DRL agent at each decision time n n The instantaneous reward function can be calculated using the following formula:

[0103] R n =β clr ·M clarity (n)+β prot ·M protection (n)-β load ·P overload (Lcog (n))+β comm ·M comm_eff (n)-β ctrl ·P action_cost (A n );

[0104] In the formula, R n M represents the instantaneous reward value obtained at decision time n; clarity (n) represents a quantitative indicator of the target speech clarity at decision time n, which can be calculated, for example, by objective speech quality assessment algorithms (such as STOI, PESQ); M protection (n) represents a quantitative indicator of the effectiveness of hearing protection for the pilot at decision time n, such as assessing whether the equivalent A-weighted sound pressure level of the output audio is below a preset safety threshold; L cog (n) represents the pilot's cognitive state at decision time n (the inference result from step S3); P overload (L cog (n) represents a value based on the current pilot's cognitive load L. cog (n) The calculated overload penalty function increases when the cognitive load is high to avoid introducing excessive auditory stimulation; M comm_eff (n) represents a quantitative indicator of communication efficiency at decision time n, which can be evaluated based on recognition accuracy or the completeness of information transmission; A n It is the sound field modulation action (i.e., a specific set of modulation parameters) output by the DRL agent at decision time n; P action_cost (A n ) represents the sound field modulation action A selected by the current DRL agent. n The penalty function applied to cost or smoothness is used to avoid overly drastic parameter adjustments that could lead to abrupt or unstable sound; while β clr ,β prot ,β load ,β comm ,β ctrl These are pre-set weighting coefficients used to balance the importance of each component in the instantaneous reward function. These weights can be adjusted according to specific application scenarios.

[0105] The DRL agent (e.g., employing value-based algorithms such as Deep Q-Network (DQN), policy-based algorithms such as A3C, or Actor-Critic type algorithms) makes decisions and outputs a set of specific sound field modulation parameters based on its learned optimal action strategy after receiving inputs such as the current pilot state and the importance of the speech. These parameters are then used to guide personalized enhancement processing of the target speech source separated in step S2. For example, based on the pilot's attention focus and the semantic importance of the speech, parameters such as gain, frequency equalization, and dynamic range compression of the target speech are adjusted to improve its intelligibility and saliency. Simultaneously, these parameters are also used to guide adaptive suppression processing of the environmental noise source.

[0106] In one possible implementation, adaptive suppression of the ambient noise source is achieved through a Model-Predictive-Control (MPC) framework. This MPC framework is specifically optimized for active control of ambient noise. The MPC framework utilizes short-term predictive information obtained by analyzing the characteristics of the ambient noise sources separated in step S2 (e.g., their spectral structure, statistical properties, and short-term trends), and combines this with the pilot's attention flow information inferred in step S3 (e.g., if the pilot is not paying attention to a particular noise source direction, the suppression of noise in that direction can be increased), to optimize the generation of anti-phase signals in the Active-Noise-Reduction (ANR) system.

[0107] The core objective of the MPC framework is to minimize a cost function within a predefined prediction-horizon. The model's prediction control cost function J... MPC It can be represented as:

[0108]

[0109] In the formula, J MPC t0 represents the cost function value calculated by Model Predictive Control at the current decision time; t0 represents the current discrete time point of MPC control; and i represents the step size index in the prediction time domain, ranging from 0 to P. mpc -1, where P mpc It is the length of the prediction time domain (i.e., the number of forward prediction steps); y res (t0+i|t0) represents the predicted state vector of the residual noise (i.e., the result of the superposition of the original ambient noise and the actively generated anti-phase sound signal) at the pilot's headset at the current time t0 for the i-th future step; y ref(t0+i|t0) represents the expected residual noise target state vector at the i-th future step, set at the current time t0. Ideally, this target vector would be zero, or it could be set to a non-zero target if the pilot's attention allows for a specific background noise level. anr (t0+i) represents the anti-phase signal control vector decided at the current time t0 and planned to be applied to the speaker of the active noise cancellation system in the i-th step in the future; Q mpc R is a positive definite weight matrix used to penalize the deviation between the predicted residual noise state vector and the desired target state vector; mpc It is a positive definite weight matrix used to penalize the energy of the anti-phase signal control vector to avoid excessive control output or unstable system behavior. By solving this optimization problem online in each control cycle, MPC can compute a series of optimal control inputs u. anr The system typically applies the first control input of the sequence to the system and then repeats the process at the next time step, thereby achieving dynamic and forward-looking suppression of environmental noise.

[0110] After personalizing the target speech source and adaptively suppressing ambient noise sources, the system mixes these independently processed sound source signals and synthesizes them into a binaural audio signal. Binaural synthesis techniques, such as utilizing the Head Relational Transfer Function (HRTF) or simplified binaural cues (e.g., interaural time difference (ITD) and interaural intensity difference (ILD), are employed to preserve or reconstruct the spatial orientation of the sound source as much as possible, enabling the pilot to perceive the direction of the sound source through hearing—crucial for maintaining situational awareness. Finally, the synthesized binaural audio signal is played back through the pilot's headphones.

[0111] Please see Figure 2 The present invention also provides a voice recognition compensation-based hearing protection system for pilot headsets, the system comprising the following modules:

[0112] The multimodal perception data acquisition module is configured to collect multi-channel acoustic signals from the pilot's surrounding environment and simultaneously acquire flight parameters to form multimodal perception data.

[0113] This module serves as the data input front-end of the entire system, its core function being to comprehensively and in real-time capture various information related to the pilot and their acoustic and flight environment. Specifically, this module is equipped with multiple microphone channels, preferably deployed in an array at predetermined locations within the pilot's helmet or cockpit, to collect multi-channel acoustic signals from the pilot's surroundings. These signals can capture the pilot's own voice, conversations from air traffic control or other crew members, and various environmental noises such as engine roar and airflow noise, while preserving the spatial information of the sound sources. Simultaneously, the module synchronously acquires key flight parameters, such as aircraft altitude, speed, attitude, and engine status, through data interfaces with the onboard avionics system or dedicated sensors. In some preferred embodiments, the module can be further configured to collect the pilot's head posture data (e.g., head orientation obtained through an inertial measurement unit built into the helmet) or physiological state data (e.g., monitoring heart rate, skin conductance, etc., through wearable sensors). All collected data is precisely timestamped and synchronized, collectively forming rich and time-consistent multimodal sensing data, laying the foundation for subsequent analysis and processing.

[0114] The sound source decoupling module is configured to use a tensor decomposition-based sound source decoupling algorithm to analyze the acoustic scene of the multi-channel acoustic signals in the multimodal sensing data, thereby separating the target speech sound source and the environmental noise sound source.

[0115] This module receives multi-channel acoustic signals from a multimodal sensing data acquisition module and applies an advanced sound source decoupling algorithm for in-depth analysis. The aim is to accurately separate individual sound source components from complex mixed acoustic scenes. Specifically, this module employs a tensor decomposition-based sound source decoupling algorithm. First, the time-domain multi-channel acoustic signals are converted to the time-frequency domain and constructed into a high-order observation signal tensor. Then, using a specific tensor decomposition model (such as normalized multivariate regression / parallel factor analysis CPD / PARAFAC), this observation tensor is decomposed into the sum of several rank-tensors. Each rank-tensor represents a potential independent sound source and carries its unique temporal activation pattern, frequency characteristics, and spatial characteristics (such as azimuth information relative to the microphone array). By analyzing and reconstructing the individual sound source factors obtained from the decomposition, this module can effectively separate clear target speech sources (such as a pilot's speech or important external commands) from various types of environmental noise sources.

[0116] The pilot state inference module is configured to input the flight parameters, the acoustic characteristics of the target speech source and the environmental noise source in the multimodal perception data into a pre-trained dynamic attention network model to infer the pilot's current attention flow and cognitive state.

[0117] The core task of this module is to use an artificial intelligence model to infer the pilot's current internal state, particularly their attention flow and cognitive load level, from multimodal perception data. Specifically, this module uses flight parameters acquired by the multimodal perception data acquisition module, and the acoustic characteristics (such as energy, spectrum, and spatial location) of the target speech source and environmental noise source separated by the sound source decoupling module, as input features, and feeds them into a pre-trained dynamic attention network model. This network model, for example, may contain graph neural network layers and recurrent neural network layers, capable of learning the complex relationships between these heterogeneous input data and capturing their dynamic patterns over time. Graph neural network layers can effectively fuse information from different sources and represent their interrelationships, while recurrent neural network layers excel at processing sequential data and understanding the evolution of the pilot's state. Through this model's processing, it can output a judgment of the acoustic object or direction in which the pilot's current attention is primarily focused, as well as an assessment of their cognitive stress, fatigue level, and other states.

[0118] The speech processing and semantic evaluation module is configured to determine the target speech source for the pilot's attention flow, combine the flight parameters in the multimodal perception data as context to perform context-aware automatic speech recognition to obtain the recognized text, and evaluate the semantic importance of the recognized text based on the aviation domain knowledge graph to obtain the semantic importance result.

[0119] This module focuses on the refined processing and understanding of the target speech source that the pilot's attention is directed towards. Once the pilot state inference module identifies the target speech source, this module first performs context-aware automatic speech recognition (ASR) on that source. During speech recognition, it relies not only on the acoustic signal itself but also incorporates flight parameters from multimodal sensing data (such as current flight phase, altitude, and speed) as crucial contextual information. This guides the ASR model in selecting more appropriate acoustic and language models, thereby improving accuracy and robustness in noisy environments and ultimately converting the speech content into recognized text. Subsequently, this module utilizes a pre-built aviation knowledge graph to assess the semantic importance of the acquired recognized text. This knowledge graph stores aviation terminology, standard instructions, emergency symbols, and other information. By matching and semantically analyzing the recognized text against entries in the knowledge graph, the module can determine the criticality and priority of the speech content in the current flight context and output corresponding semantic importance results.

[0120] The sound field modulation and output module is configured to, based on the pilot's attention flow, cognitive state and semantic importance results, use a sound field modulation strategy based on deep reinforcement learning to personalize the target speech source and adaptively suppress the environmental noise source, and then synthesize the processed target speech source and environmental noise source into a binaural audio signal and output it to the pilot's headphones.

[0121] This module is the final execution unit of the system, responsible for dynamically optimizing the sound heard by the pilot based on the processing results of the preceding modules. Specifically, it applies a sound field modulation strategy based on deep reinforcement learning (DRL), taking into account the pilot's attention flow and cognitive state output by the pilot state inference module, as well as the semantic importance results obtained by the speech processing and semantic evaluation module. This DRL agent learns and autonomously decides how to adjust sound field parameters to achieve the best auditory effect. Specifically, the DRL strategy guides personalized enhancement of target speech sources with high semantic importance, determined by the pilot's attention flow, such as improving their clarity, adjusting their loudness or spatial representation to make them easier to perceive and understand. Simultaneously, for sound sources identified as environmental noise, adaptive suppression is performed, for example, through advanced active noise reduction techniques such as model predictive control (MPC), dynamically adjusting the generation of anti-phase signals according to the characteristics of the noise and the pilot's state to minimize noise interference and protect the pilot's hearing. The target speech source and the ambient noise source (or their remnants), after enhancement and suppression processing, will be intelligently combined into a binaural audio signal. This synthesis process will try to preserve or reconstruct the spatial sense of the sound source, and then output through the pilot's headphones to provide a clear, comfortable, and information-layered personalized auditory environment.

[0122] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for hearing protection in pilot headsets with voice recognition compensation, characterized in that, The method includes the following steps: S1. Collect multi-channel acoustic signals of the pilot's surrounding environment and simultaneously acquire flight parameters to form multimodal sensing data; S2. Apply a sound source decoupling algorithm based on tensor decomposition to analyze the acoustic scene of the multi-channel acoustic signals in the multimodal sensing data, and separate the target speech sound source and the environmental noise sound source. S3. Input the flight parameters, the acoustic characteristics of the target speech source and the environmental noise source in the multimodal perception data into the pre-trained dynamic attention network model to infer the pilot's current attention flow and cognitive state. S4. For the target speech source whose pilot attention flow is determined, combine the flight parameters in the multimodal perception data as context to perform context-aware automatic speech recognition to obtain the recognition text, and evaluate the semantic importance of the recognition text based on the aviation domain knowledge graph to obtain the semantic importance result. S5. Based on the pilot's attention flow, cognitive state, and semantic importance, a sound field modulation strategy based on deep reinforcement learning is applied to personalize the target speech source and adaptively suppress the environmental noise source. The processed sound source is then synthesized into a binaural audio signal and output to the pilot's headphones.

2. The method for hearing protection of pilot headsets with voice recognition compensation according to claim 1, characterized in that, In step S1, the process of collecting multi-channel acoustic signals from the pilot's surrounding environment and simultaneously acquiring flight parameters to form multimodal perception data also includes collecting the pilot's head posture data or physiological state data.

3. The method for hearing protection of pilot headsets with voice recognition compensation according to claim 1, characterized in that, In step S2, the step of applying a source decoupling algorithm based on tensor decomposition to analyze the acoustic scene of the multi-channel acoustic signal includes: The multi-channel acoustic signals are transformed from the time domain to the time-frequency domain and constructed into a time-frequency domain observation signal tensor. In the formula, K represents the number of time frames, F represents the number of frequency points, and M represents the number of frequency points. total Indicates the total number of microphone channels; The observed signal tensor was analyzed using a normalized multiple regression model. The sound sources are decomposed to separate them; this decomposition is expressed by the CPD model formula: In the formula, N src λ represents the estimated number of independent sound sources; r is the index of the sound source; λ r Represents the complex amplitude of the r-th sound source; This represents the column vector of time activation modes for the r-th sound source. This represents the frequency response column vector of the r-th sound source; The column vector representing the microphone or spatial factor of the r-th sound source; Represents the outer product operation of tensors; The factors of each sound source obtained based on the CPD model decomposition, namely and The target speech source and the environmental noise source are reconstructed separately.

4. The method for hearing protection of pilot headphones with voice recognition compensation according to claim 1, characterized in that, In step S3, the pre-trained dynamic attention network model includes a graph neural network layer and a recurrent neural network layer with subsequent connections, used to process the acoustic characteristics of the flight parameters, the target speech source and the environmental noise source. The pre-trained dynamic attention network model further infers the pilot's current attention flow and cognitive state based on the pilot's head posture data or physiological state data. The GNN layer constructs graph-structured data from the flight parameters, the acoustic characteristics of the target speech source, and the environmental noise source. Different information entities serve as nodes in the graph, and a graph attention mechanism is applied to learn the relationships between the nodes to update the representation of each node. After the update of any target node v in time frame n, the node representation h′ is obtained. v,n Calculation using the formula updated via GNN nodes: In the formula, h′ v,n Let σ be the node representation of target node v after the update in time frame n; GNN The non-linear activation function of the GNN layer is used; the summation iterates through all source nodes u, where the source nodes u are selected from the set of neighbor nodes of the target node v. Or the target node v itself; α vu,n W represents the attention coefficient of the source node u to the target node v at time frame n, calculated using the attention mechanism. GNN h is the learnable weight matrix of the GNN layer; u,n This is the input feature vector of the source node u in time frame n; The RNN layer receives the updated node representation output by the GNN layer and combines it with temporal information to capture dynamic change patterns through the transmission of internal network states, ultimately outputting results that characterize the pilot's attention flow and cognitive state.

5. The method for hearing protection of pilot headsets with voice recognition compensation according to claim 1, characterized in that, In step S4, when processing the target speech source determined by the pilot's attention flow, the target speech source, together with the flight parameters in the multimodal perception data, is used as context information and input into a context-aware automatic speech recognition model to perform speech recognition operations and generate corresponding recognized text. The generated identified text is matched and semantically analyzed with a pre-configured aviation domain knowledge graph, which stores aviation-specific vocabulary, standard instruction structures, or identifiers representing urgency. This analysis process is used to evaluate the semantic importance of the identified text and ultimately obtain a quantified semantic importance result.

6. The method for hearing protection of pilot headphones with voice recognition compensation according to claim 1, characterized in that, In step S5, a sound field modulation strategy based on deep reinforcement learning is applied, specifically referring to: A deep reinforcement learning agent is used, which takes the pilot's attention flow, cognitive state and semantic importance as part of the input state for its decision-making, and optimizes the action strategy by interacting with the acoustic environment, aiming to maximize a predefined cumulative reward. The instantaneous reward R obtained by the DRL agent at each decision time n n Calculated using the instantaneous reward function formula: R n =b clr ·M clarity (n)+β prot ·M protection (n)-b load ·P overload (L cog (n))+β comm ·M comm_eff (n)-b ctrl ·P action_cost (A n ); In the formula, R n M is the instantaneous reward value evaluated at decision time n; clarity (n) represents the quantitative indicator of the target speech clarity at decision time n; M protection (n) represents a quantitative indicator of the effectiveness of hearing protection for the pilot at decision time n; P overload (L cog (n) represents a value based on the current pilot's cognitive load L. cog (n) The overload penalty function is calculated; M comm_eff (n) represents a quantitative indicator of communication efficiency at decision time n; P action_cost (A n ) represents the sound field modulation action A selected by the current DRL agent. n The penalty function imposed on cost or smoothness; L cog (n) represents the pilot's cognitive state; A n It is the sound field modulation action output by the DRL agent at time n; while β clr ,β prot ,β load ,β comm ,β ctrl These are the preset weighting coefficients used to balance the importance of the above indicators; The DRL agent makes decisions based on the learned action strategy and outputs a set of specific sound field modulation parameters. These parameters are used to guide the personalized enhancement processing of the target speech source and the adaptive suppression processing of the environmental noise source.

7. The method for hearing protection of pilot headsets with voice recognition compensation according to claim 1, characterized in that, In step S5, the adaptive suppression of the environmental noise source is achieved through a model predictive control framework. The MPC framework utilizes short-term prediction information obtained by analyzing the characteristics of the isolated environmental noise sources, and combines it with the inferred pilot attention flow to optimize the generation of anti-phase signals in the active noise cancellation system. The goal of the MPC framework is to minimize a cost function within a preset prediction time domain. The model predicts the control cost function J. MPC Represented as: In the formula, J MPC t0 represents the cost function value of model predictive control; t0 represents the current discrete time point of MPC control; i represents the step size index in the prediction time domain; P mpc It is the length of the prediction time domain; y res (t0+i|t0) represents the state vector of residual noise at the pilot's headset at the i-th future step predicted from the current time t0; y ref (t0+i|t0) represents the expected residual noise target state vector at the i-th future step, set at the current time t0; u anr (t0+i) represents the anti-phase signal control vector decided at the current time t0 and planned to be applied in the i-th step in the future; Q mpc R represents the weight matrix used to penalize residual noise error. mpc This represents the weight matrix used to penalize the control input energy.

8. A voice recognition-compensated hearing protection system for pilot headsets, applied to the method described in any one of claims 1-7, characterized in that, The system includes the following modules: The multimodal perception data acquisition module is configured to collect multi-channel acoustic signals from the pilot's surrounding environment and simultaneously acquire flight parameters to form multimodal perception data. The sound source decoupling module is configured to use a tensor decomposition-based sound source decoupling algorithm to analyze the acoustic scene of the multi-channel acoustic signals in the multimodal sensing data, thereby separating the target speech sound source and the environmental noise sound source. The pilot state inference module is configured to input the flight parameters, the acoustic characteristics of the target speech source and the environmental noise source in the multimodal perception data into a pre-trained dynamic attention network model to infer the pilot's current attention flow and cognitive state. The speech processing and semantic evaluation module is configured to determine the target speech source for the pilot's attention flow, combine the flight parameters in the multimodal perception data as context to perform context-aware automatic speech recognition to obtain the recognized text, and evaluate the semantic importance of the recognized text based on the aviation domain knowledge graph to obtain the semantic importance result. The sound field modulation and output module is configured to, based on the pilot's attention flow, cognitive state, and semantic importance results, use a sound field modulation strategy based on deep reinforcement learning to personalize the target speech source and adaptively suppress the environmental noise source. Then, the processed target speech source and environmental noise source are synthesized into a binaural audio signal and output to the pilot's headphones.

Images