An aviation speech generation method and system based on a generative adversarial network

Abstract

The application belongs to the technical field of data processing, and relates to an aviation speech generation method and system based on a generative adversarial network, aiming to solve the problems of weak context awareness, low accuracy of key terms, poor noise environment robustness and difficult airborne deployment of speech generation technology in the aviation field. The application comprises the following steps: obtaining a text instruction and multi-dimensional aviation context information, preprocessing to obtain standardized text and original context features; calculating a scene emergency degree, and encoding the original context features into a context vector based on the original context features; generating an initial speech signal in combination with the context vector and the standardized text; obtaining a benchmark speech signal through verification by a double-path ASR discriminator; and performing post-processing according to the scene emergency degree to generate a final speech adapted to an airborne system. Through context-aware generation and double-path collaborative verification, combined with dynamic quantization and safety constraints, the safety, scene adaptability, semantic accuracy and real-time performance of aviation speech generation are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to an aviation speech generation method and system based on a generative adversarial network. BACKGROUND

[0002] With the development of artificial intelligence technology, speech generation (Text-to-Speech, TTS) technology has been widely applied in human-computer interaction systems. In the aviation field, speech generation technology undertakes the key tasks of broadcasting flight instructions, warning information, etc., which is crucial to flight safety. However, when the existing general speech generation technology is directly applied to the aviation field, it faces some challenges.

[0003] Firstly, the existing technology cannot guarantee the absolute accuracy of the content. The general speech generation model usually takes improving the naturalness and prosody of the speech as the main optimization target, but it cannot achieve zero error rate for key terms in a specific field, such as flight altitude "FL350" and runway number "RWY01" in aviation communication. In the aviation field, any error in a number or term may lead to misjudgment and cause catastrophic accidents, so the absolute accuracy of the content is the first prerequisite that cannot be compromised.

[0004] Secondly, the existing technology lacks robustness in extreme environments. The cockpit of an airplane is filled with strong mixed noise from multiple sources such as engines and air currents, with extremely low signal-to-noise ratio. In such harsh acoustic environments, the speech signal generated by the existing technology is easily overwhelmed by noise, resulting in a significant decrease in speech intelligibility, and the pilot may not be able to clearly identify emergency warnings or critical instructions, directly threatening flight safety.

[0005] Finally, the existing technology has poor engineering adaptability to onboard devices. In order to pursue high-fidelity speech effects, advanced deep learning speech generation models usually have complex structures and huge parameter quantities, requiring high computing power and memory resources. The computing resources of onboard avionics devices are extremely limited, and the real-time requirements for task processing are extremely high (usually milliseconds). The existing model is difficult to efficiently deploy and run on the onboard platform that meets the low delay, low power consumption, and small memory requirements.

[0006] Therefore, how to provide an aviation speech generation technology that can simultaneously meet the requirements of content accuracy, extreme environment robustness, and low resource deployment is a technical problem that needs to be solved in the field. SUMMARY

[0007] In order to solve the above problems in the prior art, i.e. to solve the technical problems of low content accuracy, poor noise environment robustness, and difficulty in deploying onboard devices of the existing speech generation technology in the aviation field, the present application provides an aviation speech generation method based on a generative adversarial network, which comprises the following steps:

[0008] acquire text instructions and synchronously collect multi-dimensional aviation context information, preprocess the text instructions and the multi-dimensional aviation context information to obtain standardized text and original context features respectively;

[0009] calculate scene emergency degree based on the original context features, and encode the original context features to generate a context vector;

[0010] generate an initial voice signal by a pre-trained context perception generator in combination with the context vector and the standardized text;

[0011] collaboratively verify the initial voice signal by a double-path ASR (Automatic Speech Recognition) discriminator based on a pre-set aviation key term library to obtain a benchmark voice signal;

[0012] perform scene adaptive post-processing on the benchmark voice signal according to the scene emergency degree to generate a final voice and transmit the final voice to an airborne broadcasting system for broadcasting; wherein the post-processing includes multi-source noise separation, dynamic precision quantization and loudness adaptation.

[0013] In some preferred embodiments, the multi-dimensional aviation context information at least includes one or more of flight phase data output by a flight control system, aircraft state data, environmental noise data collected by an onboard microphone, and control sector data output by a global positioning system; and the preprocessing includes ICAO standard term standardization processing on the text instructions, time stamp synchronization and abnormal value removal on the multi-dimensional aviation context information.

[0014] In some preferred embodiments, the original context features are encoded to generate a context vector by a method comprising:

[0015] performing dynamic convolution on the original context features to extract local time sequence features;

[0016] mapping the local time sequence features to a high-dimensional vector by a dimension projection matrix;

[0017] performing nonlinear transformation on the high-dimensional vector by an activation function to generate a context vector.

[0018] In some preferred embodiments, the scene emergency degree is calculated based on the original context features by a method comprising:

[0019] analyzing the current alert level and the flight phase based on the original context features, and performing weighted fusion based on the pre-set weights corresponding to the alert level and the flight phase to obtain the scene emergency degree;

[0020] The alarm levels include no alarm, prompt alarm, and emergency alarm, and each level has different weights; the flight stages include a cruising stage, an approach stage, and a takeoff or landing stage, and each stage has different weights.

[0021] In some preferred embodiments, an initial speech signal is generated by jointly combining the context vector and the standardized text through a context-aware generator in a pre-trained generative adversarial network, by:

[0022] The standardized text is converted into a text feature vector, and the text feature vector is dimensionally aligned and fused with the context vector to obtain a joint feature;

[0023] Based on the joint feature, a context-adapted frequency spectrum is generated by a context-aware generator;

[0024] A pre-trained generative adversarial network is called to iteratively optimize the acoustic features and semantic consistency of the context-adapted frequency spectrum based on pre-training parameters;

[0025] The optimized context-adapted frequency spectrum is converted into a time-domain waveform to generate an initial speech signal.

[0026] In some preferred embodiments, a context-adapted frequency spectrum is generated by a context-aware generator based on the joint feature, by:

[0027] A base mel-frequency spectrum is generated based on the text feature vector;

[0028] Based on a gated attention mechanism, a frequency attention vector matching the dimension of the context vector is configured for each mel-frequency band, and the weight distribution of the frequency attention vector is dynamically adjusted according to the scene urgency;

[0029] The joint feature and the adjusted frequency attention vector are associated to generate a weight coefficient for each mel-frequency band;

[0030] The base mel-frequency spectrum is weighted and optimized by frequency band using the weight coefficient to obtain a context-adapted frequency spectrum.

[0031] In some preferred embodiments, an incremental update mechanism is adopted in the process of generating an initial speech signal:

[0032] The regeneration of the context vector is triggered by real-time monitoring of the original context feature change amount, and the context vector of the previous moment is used when the change amount does not reach a preset threshold.

[0033] In some preferred embodiments, the dual-path ASR discriminator includes a main-path ASR discriminator and an auxiliary-path ASR discriminator; and the initial speech signal is cooperatively verified to obtain a reference speech signal, by:

[0034] The initial speech signal is identified by using a main path ASR discriminator to generate a first identification result; wherein the main path ASR discriminator takes aviation special terms as the identification benchmark;

[0035] The initial speech signal is identified by using a secondary path ASR discriminator, and deviation compensation is performed through feature mapping and regularization optimization to generate a second identification result;

[0036] The word error rate is calculated based on the first identification result and the standardized text, and the adversarial calibration loss is calculated based on the difference between the first identification result and the second identification result;

[0037] When the word error rate is zero and the adversarial calibration loss is lower than a preset threshold, it is determined that the verification is passed, and the current speech signal is confirmed as the benchmark speech signal; otherwise, the context-aware generator is fed back and optimized in combination with the word error rate and the adversarial calibration loss, and the initial speech signal is regenerated.

[0038] In some preferred embodiments, dynamic precision quantization is performed by the following method:

[0039] The target quantization precision level is determined according to the scene emergency, and the quantization precision level includes full-model low-precision quantization, mixed-precision quantization and full-model high-precision quantization;

[0040] The target quantization precision level is applied to the acoustic model, and the benchmark speech signal is converted by the acoustic model to generate the final speech.

[0041] The second aspect of the application proposes an aviation speech generation system based on a generative adversarial network, which comprises:

[0042] An information processing module configured to receive a text instruction and synchronously collect multi-dimensional aviation context information, perform ICAO standard term standardization processing on the text instruction, perform timestamp synchronization processing on the multi-dimensional aviation context information, and output the standardized text and the original context feature;

[0043] A context generation module configured to calculate the scene emergency and encode the context vector based on the original context feature; generate an initial speech signal by combining the context vector and the standardized text through a pre-trained context-aware generator; wherein the process of generating the initial speech signal includes dynamic optimization of the mel frequency band by the gated attention mechanism and real-time adaptation of the context vector by the incremental update mechanism;

[0044] A collaborative verification module configured to perform collaborative verification on the initial speech signal based on a double-path ASR discriminator; when the verification is passed, the initial speech signal is the benchmark speech signal; and when the verification fails, a feedback signal is output to the context generation module;

[0045] The voice generation module is configured to perform scene adaptive post-processing on the reference voice signal according to a scene emergency degree, generate a final voice and transmit the final voice to an airborne broadcasting system for broadcasting.

[0046] The present application has the following advantages:

[0047] 1. The present application realizes the dynamic cognitive closed loop of "generation, verification, feedback and optimization" of aviation voice based on double-path verification and aviation terminology optimization through multi-dimensional context fusion and emergency degree driving, and the generated voice can dynamically adapt to the scene characteristics of different flight stages and noise environments, can realize zero-error generation of aviation key terminology, improve the intelligibility in extreme noise environments, and meet the requirements of airborne engineering deployment, thereby comprehensively improving the safety and reliability of aviation voice communication.

[0048] 2. The generated voice content is consistent with the semantics of the original text instruction through the cooperative verification and closed loop feedback mechanism of the double-path ASR discriminator, the main path reference recognition and the auxiliary path deviation compensation, thereby greatly improving the safety of aviation voice interaction.

[0049] 3. By real-time acquisition and fusion of multi-dimensional aviation context information, four types of dynamic information of flight stage, aircraft state, environmental noise and control sector are integrated into the generation process, and various complex and emergency scenes are matched.

[0050] 4. By adjusting the frequency band weight and matching the loss function, the initial voice adapted to the scene is generated, and through the scene adaptive post-processing, especially the dynamic precision quantization and loudness adaptation, the final output voice signal can meet the fidelity requirements under different emergency degrees, efficiently utilize the limited computing resources of the airborne system, and solve the problem of difficult deployment of high-performance models on airborne devices. BRIEF DESCRIPTION OF DRAWINGS

[0051] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0052] Figure 1 is a flowchart of an aviation voice generation method based on a generative adversarial network in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, in order to facilitate description, only the parts related to the application are shown in the drawings.

[0054] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0055] The present application provides an aviation speech generation method based on a generative adversarial network. Around the three core requirements of safety, robustness and landing of aviation speech generation, through the cooperation of a dual-path ASR discriminator and a four-dimensional context perception generator, combined with phased dynamic weight scheduling, scene perception dynamic quantization and aviation-specific safety hard constraint adaptation to aviation scene characteristics optimization, the limitations of general speech synthesis and traditional GAN (Generative Adversarial Network) technology are broken through, key term recognition, extreme noise robustness and airborne low resource adaptation are realized, and civil aviation safety and engineering standards are met.

[0056] The aviation speech generation method based on the generative adversarial network of the present application comprises the following steps:

[0057] Obtaining text instructions and synchronously collecting multi-dimensional aviation context information, preprocessing the text instructions and the multi-dimensional aviation context information to obtain standardized text and original context features respectively;

[0058] Based on the original context features, the scene emergency degree is calculated, and the original context features are encoded to generate a context vector;

[0059] Through the pre-trained context perception generator, the context vector and the standardized text are combined to generate an initial speech signal;

[0060] Based on a pre-set aviation key term library, the dual-path ASR discriminator is used to cooperatively verify the initial speech signal to obtain a reference speech signal;

[0061] According to the scene emergency degree, the reference speech signal is subjected to scene adaptive post-processing to generate a final speech and transmit it to an airborne broadcasting system for broadcasting; wherein the post-processing includes multi-source noise separation, dynamic precision quantization and loudness adaptation.

[0062] In order to more clearly illustrate the aviation speech generation method based on the generative adversarial network of the present application, the following will be described in combination with Figure 1 The steps in the embodiments of the present application will be described in detail.

[0063] The aviation speech generation method based on the generative adversarial network of the first embodiment of the present application comprises steps S1-S5, and the steps are described in detail as follows:

[0064] S1, obtaining text instructions and synchronously collecting multi-dimensional aviation context information, preprocessing the text instructions and the multi-dimensional aviation context information to obtain standardized text and original context features respectively.

[0065] Preferably, the multi-dimensional aviation context information comprises one or more of flight phase data outputted by a flight control system, aircraft state data, ambient noise data collected by an onboard microphone, and control sector data outputted by a global positioning system.

[0066] In some embodiments, the multi-dimensional aviation context information is obtained by:

[0067] flight phase data (e.g. "takeoff", "climb", "cruise", "approach", "landing", etc.) and aircraft state data (e.g. flight altitude, airspeed, attitude angle, engine parameters, alert level, etc.) outputted by a flight control system (FCS);

[0068] ambient noise data collected by an onboard microphone installed in the cockpit, which can be analyzed by spectrum analysis to obtain the intensity and frequency distribution characteristics of the noise;

[0069] control sector data obtained by a global positioning system (GPS) or a communication management unit (CMU) to identify specific regulations or customs of the current communication airspace.

[0070] If any sensor (e.g. GPS, FCS) fails, the system automatically switches to a "degraded mode": when GPS data is missing, the last valid control sector data before the failure is used in combination with an inertial navigation system (INS) to calculate temporary airspace information; when FCS data is missing, only ambient noise, control sector, and historical aircraft state data are retained, and the "aircraft state weight" in the calculation of the scene urgency is temporarily reduced to 0.2 to compensate for the impact of data loss.

[0071] It can be understood that the above-mentioned degraded mode is only one specific implementation, and when it is detected that at least one dimension of aviation context information fails, a preset degraded mode is started, which at least includes compensating for the missing information based on historical data or alternative data sources, and / or adjusting the calculation logic of the scene urgency to reduce the dependence on the missing data source. These methods are within the scope of the present application.

[0072] In the present embodiment, the preprocessing includes:

[0073] According to the standard aviation terminology library of the International Civil Aviation Organization (ICAO), the text instructions are standardized by ICAO standard terminology to generate standardized text;

[0074] For the multi-dimensional aviation context information, a uniform timestamp is applied to the data streams from different sensors to achieve synchronization, and a filtering algorithm (in the present embodiment, a median filtering algorithm) is used to remove obvious outliers or sensor glitches to generate original context features:

[0075] ;

[0076] wherein, are flight phase feature vector (such as "takeoff = [1, 0, 0, 0, 0]", covering 5 core flight phases, dimension = 5), aircraft state feature vector (containing flight altitude, airspeed, attitude angle, engine speed, etc. 8 key parameters, dimension = 8), environmental noise feature vector (containing noise intensity, center frequency, bandwidth, duration, dimension = 4), control sector feature vector (containing airspace number, control frequency, height limit, dimension = 3).

[0077] If the text instruction contains non-ICAO standard terms, the system automatically triggers the term checking module, compares the aviation emergency term library (such as ETOPS special terms), and preferentially matches the closest standard terms and marks the confidence; if the confidence is less than 95%, immediately suspend the generation process and send a term abnormality alarm to the onboard monitoring system, and continue after manual confirmation; further, the abnormal processing process of the non-standard term can be more widely defined as: when the text instruction contains non-standard terms, a preset abnormal processing process is executed, and the abnormal processing process includes at least one of: automatic correction, suspension of voice generation, or sending an alarm to an external system.

[0078] The environmental noise data is divided into three categories according to the source and processed accordingly: engine noise (mainly low frequency, 100-500Hz, high-pass filter), thunderstorm noise (wide frequency, 500-5000Hz, adaptive noise suppression), cabin equipment noise (impulse noise, median filter), to improve the accuracy of noise separation).

[0079] S2, calculate the scene emergency degree based on the original context features, and encode the original context features to generate a context vector.

[0080] Preferably, the scene emergency degree is calculated based on the original context features, and the method is:

[0081] Based on the original context features, the current alarm level and the flight phase are analyzed, and based on the preset weights corresponding to the alarm level and the flight phase, the weighted fusion is performed to obtain the scene emergency degree;

[0082] The alarm level includes no alarm, prompt alarm, and emergency alarm, and the weights of each level are different; the flight phase includes the cruising phase, the approach phase, the takeoff or landing phase, and the weights of each phase are different.

[0083] In the embodiment, in the alarm level, the weight value of no alarm is 0, the weight value of prompt alarm is 0.5, and the weight value of emergency alarm is 1.0; in the flight phase, the weight value of the cruising phase is 0.3, the weight value of the approach phase is 0.6, and the weight value of the take-off or landing phase is 0.8.

[0084] The alarm level and the flight phase feature which have the most significant influence on the urgency of the voice are preferentially adopted, and other features such as environmental noise, control sector and the like can be further integrated into the calculation through an extended model (such as a small neural network).

[0085] Further preferably, the scene urgency is calculated according to a preset proportion: scene urgency E = (weight of alarm level x ) + (weight of flight phase x ).

[0086] In the embodiment, the weight of the alarm level is 0.6, the weight of the flight phase is 0.4, and the scene urgency is divided into scenes according to a dynamic threshold:

[0087] High-urgency scene: E≥0.8 (stall / near-ground alarm)

[0088] Normal scene: 0.3≤E<0.8 (approach / way deviation)

[0089] Low-load scene: E<0.3 (cruising normal instruction).

[0090] It can be understood that the above weight and are preferred values of the embodiment, and a person skilled in the art can adjust them according to actual application requirements or through experimental data. In addition, the calculation of the scene urgency can also integrate more context features, such as combining the specific alarm signal strength in the aircraft state data, the decibel level of the environmental noise and the like, and comprehensively evaluating them through a more complex model (such as a small neural network or a fuzzy logic system), and these equivalent changes all fall within the protection scope of the present application.

[0091] Preferably, the original context features are encoded to generate a context vector, and the method is as follows:

[0092] A one-dimensional dynamic convolution network is used to process the context features in the form of time series to capture the local dynamic changes in the time dimension and extract local time sequence features; the local time sequence features are aggregated in the time dimension through global average pooling to compress the three-dimensional features into a two-dimensional vector;

[0093] The low-dimensional time sequence features are linearly mapped to a high-dimensional space through a pre-trained dimension projection matrix to form a high-dimensional vector;

[0094] ​​The high-dimensional vector is input into a nonlinear activation function (such as ReLU or GeLU) to generate a final context vector.

[0095] Further preferably, the number of channels of the one-dimensional dynamic convolutional network is adaptively adjusted according to the urgency, and when the emergency is high, the channel C = 48, and the resolution of the key feature is improved; and when the emergency is low, the channel C = 32.

[0096] Specifically, the original context feature is encoded to generate a context vector:

[0097] ;

[0098] wherein, is a one-dimensional dynamic convolutional network, is an original context feature; is a dimension projection matrix of a convolution output, and C is a channel number; GELU is an activation function, is a projection layer parameter, is a global average pooling.

[0099] The context vector is a highly condensed mathematical representation of the current entire flight scene, which contains all the conditional information for guiding the speech generator to generate a specific style of speech.

[0100] S3, through the pre-trained context-aware generator, the context vector is combined with the standardized text to generate an initial speech signal.

[0101] The pre-trained context-aware generator is obtained based on a pre-training corpus, and the model parameters are optimized through adversarial training of the generator and the discriminator in the pre-training stage. The generative network is trained and learned through the pre-training corpus, and the pre-training corpus is 100,000 real aviation speeches (covering American / English / Chinese English accents).

[0102] Preferably, through the context-aware generator in the pre-trained generative adversarial network, the context vector is combined with the standardized text to generate an initial speech signal, and the method is:

[0103] The standardized text in step S1 is converted into a high-dimensional text feature vector through a text encoder (such as BERT), and the text feature vector is dimensionally aligned and fused with the context vector in step S2 to obtain a joint feature containing semantic information and scene information;

[0104] Based on the joint feature, the context-aware generator (such as a Transformer decoder) generates a context-adapted frequency spectrum based on the aviation scene speech mapping rule learned in the pre-training stage, and dynamically adjusts the frequency attention vector weight according to the scene urgency.

[0105] Call the pre-trained generative adversarial network, perform local weight fine-tuning based on the pre-trained parameters to optimize the spectrum, and iteratively optimize the acoustic features and semantic consistency of the context-adapted spectrum; the parameters are network parameters that have been optimized through the adversarial training of the generator and discriminator in the pre-training stage;

[0106] Precisely convert the optimized context-adapted spectrum back to the time-domain waveform through a neural vocoder (such as HiFi-GAN) from the frequency domain to generate an initial speech signal.

[0107] Further preferably, based on the joint features, generate a context-adapted spectrum through a context-aware generator, and the method is as follows:

[0108] Generate a basic Mel spectrum based on the text feature vector;

[0109] Based on the gated attention mechanism, configure a frequency attention vector that matches the dimension of the context vector for each Mel band, and dynamically adjust the weight distribution of the frequency attention vector according to the scene urgency;

[0110] Perform a dot product correlation operation on the joint features and the adjusted frequency attention vector to generate the weight coefficients for each Mel band;

[0111] Use the weight coefficients to perform per-band weighted optimization on the basic Mel spectrum to obtain the context-adapted spectrum.

[0112] Find the appropriate frequency bands through the attention vector and convert the matching degree into weights, which can achieve the adaptation of the speech spectrum to the current aviation scenario. For example, when the scene switches from "cruise" to "emergency warning", the context vector will be updated in real time (the value of the emergency warning dimension increases), thereby guiding the rapid adjustment of the weight coefficients.

[0113] Specifically, based on the gated attention mechanism, configure a frequency attention vector that matches the dimension of the context vector for each Mel band (one for each Mel band, 64-dimensional), and dynamically adjust the weight distribution of the frequency attention vector according to the scene urgency, and the method is as follows:

[0114] When E≥0.8, through the emergency coefficient Enhance the weight of the high-penetration frequency band of 2000 - 4000Hz (this frequency band has the highest recognition rate for emergency warnings), and at the same time compress the weight of the low-frequency band below 500Hz; when 0.3≤E<0.8, balance the weights of the full frequency band of 300 - 3000Hz, and when 0.3<E, only slightly enhance the weight of the frequency band of 1000 - 3000Hz.

[0115] In some embodiments, 。

[0116] Specifically, in the process of generating the initial speech signal, an incremental update mechanism is adopted:

[0117] The system continuously monitors the change amount of the context features in real time. When the change amount of any context feature exceeds a preset threshold, the regeneration of the context vector is triggered.

[0118] When the change amount does not reach the preset threshold, the context vector of the previous moment is used.

[0119] Preferably, in the present embodiment, the change amount is calculated by cosine similarity.

[0120] The preset threshold includes: the change of flight altitude exceeds 500 feet, the change of alarm level, the change of environmental noise intensity ≥ 15dB, the change of aircraft attitude angle ≥ 5°, and any one of the conditions triggers the regeneration of the context vector.

[0121] In some embodiments, the pre-training is through the adversarial training of the generator and the discriminator, and the method is:

[0122] The discriminator outputs the adversarial loss by evaluating the difference between the context adaptation spectrum and the real aviation speech spectrum. The generator is based on the adversarial loss back propagation, iteratively optimizes the acoustic features (spectrum envelope, fundamental frequency) and semantic consistency (the edit distance between the text of the generated speech and the standardized text) of the context adaptation spectrum, and dynamically adjusts the loss function weight of the generator according to the scene urgency.

[0123] The discriminator includes a global discriminator and a spectrum discriminator. The global discriminator scores the initial speech signal and the real speech respectively, and outputs the global loss The spectrum discriminator scores the base mel spectrum and the target mel respectively, and outputs the spectrum loss The global loss and the spectrum loss are weighted to obtain the adversarial loss: ;

[0124] The loss function of the generator The weight is dynamically adjusted with the urgency:

[0125] ;

[0126] Wherein, is the spectrum reconstruction loss, which is generated based on the L1 loss of the base mel spectrum and the target mel combined with the spectrum cone loss (which is used to ensure that the energy of the generated spectrum in a certain key frequency band does not be lower than a preset safety lower limit, so as to ensure the basic intelligibility of the speech), which is used to constrain the spectrum structure of the generated speech to be consistent with the target speech, is the linguistic consistency loss, which is generated based on the ASR recognition result of the generated speech and the standardized text, which is used to ensure the semantic content of the generated speech to be consistent with the input text, For safety constraint loss, the scene urgency and the term recognition confidence are used to generate.

[0127] Specifically, the safety constraint total loss can be expressed as:

[0128] ;

[0129] ;

[0130] ;

[0131] ;

[0132] wherein the sub-loss weight coefficient , is the key safety term set; is the safety loss, is the grade weight, is the safety term grade (according to the aviation safety grade, it is divided into three grades: fatal, emergency, and normal, each grade is given different weights, decreasing in turn); is the term set corresponding to the grade; is the minimum confidence threshold of the corresponding grade term, is the recognition probability of the key term by the main path ASR discriminator; is the loss term of a single term, which only generates loss when the recognition probability is lower than the threshold (the greater the difference, the greater the loss), and the loss is 0 within the threshold to avoid excessive constraint;

[0133] is the time decay loss, which gives higher weight to newly appeared safety terms (such as sudden alarms) for dynamically updated contexts, and introduces a time decay factor , , is the first appearance time of the term; is the decay constant, which is 5s in this embodiment; is the set of newly added safety terms in the past 5 seconds, which needs to be prioritized to ensure real-time performance; The threshold of P1 term is reused, because the newly added terms are mostly sudden instructions in emergency scenes, which need to be constrained according to the highest standard;

[0134] is the dual-path consistency loss, which introduces the recognition difference penalty of the main and auxiliary path ASR to avoid misjudgment of a single model, is the maximum allowed difference threshold, is the recognition probability of the key term by the auxiliary path ASR discriminator; The absolute difference of the recognition probability of the primary path and the secondary path reflects the consistency between the models, and only when the difference exceeds the threshold value does a loss occur, and there is no penalty within the threshold value, balancing the strictness of the check and the reasonable deviation of the model; in the embodiment is 0.02.

[0135] The above dynamic weight is the optimal selection, the fixed spectral loss weight ensures the basic audibility, and the dynamic adjustment of the weight of other items adapts to the scene difference, and finally realizes the optimal speech generation strategy under different urgency. In the low-urgency scene (such as cruising), the balance between “realism” and “semantic accuracy” is achieved, so that the speech is natural and easy to understand; in the high-urgency scene (such as alarm): forced priority is given to “safety terminology accuracy” and “semantic consistency”, and the pursuit of “naturalness” is weakened, which is completely matched with the core principle of “safety is always the first criterion” in the aviation field.

[0136] The weight of the spectral reconstruction loss is fixed at 0.2, that is, even if the semantic accuracy is high, if the spectral distortion (such as frequency offset exceeding the receiving range of the onboard microphone), it will still cause information loss; the fixed weight ensures that the acoustic stability is not affected by the scene fluctuation.

[0137] S4, based on the preset aviation key term library, the initial speech signal is checked by the dual-path ASR discriminator to obtain a reference speech signal.

[0138] Preferably, the dual-path ASR discriminator includes a primary path ASR discriminator and a secondary path ASR discriminator; the initial speech signal is checked to obtain a reference speech signal, and the method is:

[0139] The initial speech signal is identified by the primary path ASR discriminator to generate a first recognition result; wherein the primary path ASR discriminator takes aviation-specific terms as the recognition reference;

[0140] The initial speech signal is identified by the secondary path ASR discriminator, and the deviation is compensated by feature mapping and regularization optimization to generate a second recognition result;

[0141] Based on the first recognition result and the standardized text, the word error rate is calculated, and the difference between the first recognition result and the second recognition result is compared, and the adversarial calibration loss is calculated, which reflects the acoustic clarity and unambiguousness of the speech signal;

[0142] When the word error rate is zero and the adversarial calibration loss is lower than the preset threshold value, it is determined that the check is passed, and the current speech signal is confirmed as the reference speech signal;

[0143] Otherwise, combine the word error rate and the adversarial calibration loss as the feedback signal, fine-tune the parameters of the context-aware generator through the backpropagation algorithm, and return to step S3 to regenerate a better initial speech signal. This process is repeated until the check is passed.

[0144] The main path ASR discriminator is pre-trained based on an aviation-specific corpus, and the fixed base weight ensures the stability of term recognition, establishing a term recognition benchmark. The auxiliary path ASR discriminator is based on the error rate distribution of "non-standard accent terms" in the aviation scene, and introduces a mixed effect Poisson regression layer to learn the pronunciation variant features of different genders / accent, compensating for demographic bias. The auxiliary path continuously calibrates and collaboratively supervises the output of the main path, minimizing the difference between the two paths through adversarial calibration loss to balance accuracy and robustness.

[0145] Preferably, if the main path ASR discriminator fails (such as model parameter damage), the system automatically activates the auxiliary path "main mode":

[0146] Set the bias compensation factor of the auxiliary path to 1, directly use the auxiliary path recognition result as the check benchmark, and send a "main path failure" alarm to the onboard maintenance system; if the auxiliary path fails, the main path is checked alone, and the adversarial calibration loss threshold is temporarily relaxed to 0.2 to ensure that the check process is not interrupted.

[0147] In some embodiments, the dual-path ASR discriminator performs collaborative verification, specifically:

[0148] 1) The main path ASR discriminator uses a CNN-LSTM architecture as a term recognition probability model, with the input being a speech segment (T = time step, F = frequency feature dimension), CNN extracts short-time spectral features, and LSTM captures long-time dependencies to output term recognition probability:

[0149] ;

[0150] where, The output dimension is (batch size, is the time step after convolution), is the weight matrix from CNN to LSTM, the number of channels of CNN C = 128, and the hidden layer dimension of LSTM H = 256; is the weight matrix from LSTM to the term vocabulary (V = 5000, including ICAO terms); 、 is the bias term, is the term in the vocabulary (such as "FL350" "RWY01");

[0151] 2) The auxiliary path processes the initial speech signal X independently, extracts acoustic features, calculates bias compensation factors to obtain bias features, and then outputs calibrated recognition probabilities based on the term reference of the main path:

[0152] ;

[0153] ;

[0154] wherein, is the bias compensation factor, is the exponential function; is the trainable parameter, corresponding to the weight coefficient of gender feature and accent feature respectively, is the feature discretization function, is the acoustic feature, is the random noise term; is the term recognition probability of the main path for the initial speech X, is the calibrated recognition probability;

[0155] 3) Calculate the word error rate:

[0156] ;

[0157] wherein, S is the number of substitution errors, D is the number of deletion errors, I is the number of insertion errors; is the total number of term units of the reference text (not the total number of characters, to avoid errors in short texts);

[0158] 4) Compare the difference between the recognition probabilities before and after calibration to calculate the adversarial calibration loss :

[0159] ;

[0160] wherein, is the KL divergence of the recognition probabilities before and after calibration, is the absolute difference between the main path word error rate and the auxiliary path word error rate ; is the cross-validation determined weight;

[0161] 5) When the word error rates are both zero and the adversarial calibration loss is lower than the preset threshold, it is determined that the verification passes. In this embodiment, the preset threshold is 0.1, i.e. , the dual-path cooperation is optimal.

[0162] Preferably, the auxiliary path extracts acoustic features based on Mel-spectrogram or MFCC, focusing on capturing acoustic features of gender (fundamental frequency difference: male 85-180Hz, female 165-255Hz), accent (e.g. "schedule", British pronunciation , American pronunciation ) and the like. The accent feature is judged by phoneme duration difference, and is calculated independently based on the initial voice X, without relying on the output of the main path. This dual verification mechanism of content and acoustics provides double insurance for the accuracy and robustness of the generated voice; more importantly, the signal of verification failure will fine-tune the generator parameters through a feedback mechanism, forming a closed loop of continuous optimization.

[0163] S5, performing scene adaptive post-processing on the reference voice signal according to the scene urgency, generating a final voice and transmitting it to an onboard broadcast system for broadcast.

[0164] Preferably, dynamic precision quantization is used, and the method is as follows:

[0165] According to the scene urgency, determine the target quantization precision level, which includes full model low precision quantization, mixed precision quantization and full model high precision quantization;

[0166] Apply the target quantization precision level to the acoustic model, and convert the reference voice signal through the acoustic model to generate a final voice.

[0167] Further preferably, the post-processing includes:

[0168] Multi-source noise separation: a lightweight deep learning noise reduction model is used to process the reference voice signal, further eliminating possible residual weak background noise or model generated noise, and improving the purity of the voice;

[0169] Dynamic precision quantization: the system selects different quantization strategies according to the scene urgency to adapt to the computing power of the onboard system; in high urgency scenarios (e.g. "wind shear" warning), the system uses full model high precision quantization to ensure voice details and response speed; in low urgency scenarios (e.g. routine state reporting), the system uses full model low precision quantization to maximize the saving of computing resources; in intermediate states, mixed precision quantization can be used.

[0170] Loudness adaptation: the system dynamically adjusts the loudness of the final output voice according to the real-time environmental noise level in the context features; in the take-off stage with high cockpit noise, the broadcast volume is automatically increased; in the quiet cruising stage, the volume is appropriately reduced to ensure that the voice is always clear and audible without causing interference.

[0171] Specifically, the quantization precision level is determined according to the scene emergency degree, high emergency scene is preferentially guaranteed for real-time, and low emergency scene is preferentially guaranteed for sound quality:

[0172] High emergency degree (>=0.8): INT8 full model low precision quantization is adopted (adapt to the real-time response requirement of the airborne system, delay <=50ms);

[0173] Medium emergency degree (0.3<=E<0.8): INT4 (context encoding layer) and FP8 (other layers) mixed precision quantization is adopted (balance efficiency and sound quality, delay <=100ms);

[0174] Low emergency degree (E<0.3): FP16 full model high precision quantization is adopted (preferentially guarantee sound quality, delay <=200ms);

[0175] Meanwhile, the quantization strategy is adjusted according to the computing power of the airborne system: small aircraft (such as general aircraft) with computing power <10 TOPS are forced to adopt INT8 full model quantization (regardless of the scene emergency degree); medium aircraft with computing power 10-50 TOPS are executed according to the standard emergency degree strategy; large passenger aircraft with computing power >50 TOPS can select FP32 quantization in a low emergency scene (to further improve sound quality);

[0176] The quantization precision level is applied to the acoustic model to convert the reference speech signal to generate a speech feature adapted to the computing power of the airborne system.

[0177] It should be noted that the calculation method of the full model quantization is prior art, which is not described here; the above specific computing power classification (TOPS) is only an example, a reference quantization strategy is selected according to the hardware computing power level of the airborne system carrying the method; and the quantization precision is dynamically adjusted based on the reference quantization strategy and the real-time calculated scene emergency degree. This dynamic adjustment idea is not limited by the specific computing power value and is within the protection scope of the present application.

[0178] Although the steps are described in the above order in the above embodiment, those skilled in the art can understand that, in order to achieve the effect of the embodiment, the steps do not have to be executed in such an order, they can be executed simultaneously (in parallel) or in a reversed order, and these simple changes are within the protection scope of the present application.

[0179] The second embodiment of the present application is an aviation speech generation system based on a generative adversarial network, which comprises:

[0180] The information processing module is configured to receive a text instruction and synchronously collect multi-dimensional aviation context information, perform ICAO standard term standardization processing on the text instruction, perform timestamp synchronization processing on the multi-dimensional aviation context information, and output standardized text and original context characteristics;

[0181] The context generation module is configured to calculate a scene emergency degree and encode a context vector based on the original context characteristics, and generate an initial speech signal by combining the context vector and the standardized text through a pre-trained context perception generator; wherein the process of generating the initial speech signal comprises dynamic optimization of a mel frequency band through a gated attention mechanism and real-time adaptation of the context vector through an incremental update mechanism.

[0182] The collaborative verification module is configured to perform collaborative verification on the initial speech signal based on a dual-path ASR discriminator, and when the verification is passed, the initial speech signal is a reference speech signal, and when the verification fails, a feedback signal is output to the context generation module.

[0183] The speech generation module is configured to perform scene adaptive post-processing on the reference speech signal according to the scene emergency degree, generate a final speech, and transmit the final speech to an airborne broadcasting system for broadcasting; wherein the post-processing process comprises:

[0184] The multi-source noise separation unit is used to eliminate the interference of environmental noise on the quality of the speech; the dynamic precision quantization unit selects a target quantization precision level according to the scene emergency degree, and converts the reference speech through an acoustic model to optimize the details of the speech; and the loudness adaptation unit adjusts the loudness of the speech according to the broadcasting environment of the airborne system.

[0185] It should be noted that the above embodiment provides an aviation speech generation system based on a generative adversarial network, and only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the modules or steps in the embodiment of the present application are further decomposed or combined, for example, the modules of the above embodiment can be combined into one module, or can be further split into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present application are only for distinguishing the modules and steps, and should not be considered as an improper limitation of the present application.

[0186] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related description of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0187] The third embodiment of the electronic device of the present application comprises:

[0188] at least one processor; and

[0189] a memory in communication with the at least one processor; wherein

[0190] The memory stores instructions executable by the processor for implementing the above-described method for generating aviation speech based on a generative adversarial network.

[0191] A computer-readable storage medium of the fourth embodiment of the present application, the computer-readable storage medium stores computer instructions for being executed by a computer to implement the above-described method for generating aviation speech based on a generative adversarial network.

[0192] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes and related descriptions of the electronic device and the computer-readable storage medium described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0193] Those skilled in the art should appreciate that the modules, method steps of each example described in conjunction with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. The software modules, method steps corresponding to the program can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been generally described in the foregoing description. Whether the functions are performed by electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0194] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0195] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0196] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. For example, singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0197] The description and drawings are illustrative and are not intended to be limiting. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is available. It is intended that the scope of the application encompass the various combinations of the elements disclosed above.

[0198] The technical scheme of the present application has been described in combination with the preferred embodiments shown in the drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical schemes after the changes or replacements will all fall within the protection scope of the present application.

Claims

1. An aerial speech generation method based on a generative adversarial network, characterized in that, The method comprises the following steps: acquiring a text instruction and synchronously collecting multi-dimensional aviation context information, preprocessing the text instruction and the multi-dimensional aviation context information to obtain standardized text and original context features respectively; calculating a scene emergency degree based on the original context features and encoding the original context features to generate a context vector; generating an initial voice signal by a pre-trained context perception generator in combination with the context vector and the standardized text; performing collaborative verification on the initial voice signal by a double-path ASR discriminator based on a pre-set aviation key term library to obtain a benchmark voice signal; the double-path ASR discriminator comprises a main-path ASR discriminator and an auxiliary-path ASR discriminator; the collaborative verification on the initial voice signal to obtain the benchmark voice signal is performed in the following manner: identifying the initial voice signal by the main-path ASR discriminator to generate a first identification result; wherein the main-path ASR discriminator takes aviation-specific terms as the identification benchmark; identifying the initial voice signal by the auxiliary-path ASR discriminator, performing deviation compensation through feature mapping and regularization optimization to generate a second identification result; calculating a word error rate based on the first identification result and the standardized text and calculating an adversarial calibration loss based on the difference between the first identification result and the second identification result; when the word error rate is zero and the adversarial calibration loss is lower than a pre-set threshold, determining that the verification is passed and confirming that the current voice signal is the benchmark voice signal; otherwise, feeding back the word error rate and the adversarial calibration loss to optimize the context perception generator and regenerate the initial voice signal; performing scene adaptive post-processing on the benchmark voice signal according to the scene emergency degree to generate a final voice and transmitting the final voice to an onboard broadcasting system for broadcasting; wherein the post-processing comprises multi-source noise separation, dynamic precision quantization and loudness adaptation. 2.The method of claim 1, wherein, The multi-dimensional aviation context information at least comprises one or more of flight phase data output by a flight control system, aircraft state data, environmental noise data collected by an onboard microphone and control sector data output by a global positioning system; the preprocessing comprises ICAO standard term standardization processing on the text instruction, time stamp synchronization and abnormal value removal on the multi-dimensional aviation context information. 3.The method of claim 1, wherein, The method for encoding the original context features to generate a context vector is as follows: performing dynamic convolution on the original context features to extract local time sequence features; mapping the local time sequence features into a high-dimensional vector through a dimension projection matrix; performing non-linear transformation on the high-dimensional vector by an activation function to generate a context vector. 4.The method of claim 1, wherein, The method for calculating a scene emergency degree based on the original context features is as follows: analyzing the current alarm level and the flight phase based on the original context features, performing weighted fusion based on the pre-set weights corresponding to the alarm level and the flight phase to obtain the scene emergency degree; the alarm level comprises no alarm, prompt alarm and emergency alarm, and the weights of different levels are different; the flight phase comprises a cruising phase, an approach phase, a take-off or landing phase, and the weights of different phases are different.

5. The method of claim 1, wherein the method is based on a generative adversarial network. The method for generating the initial speech signal by the context-aware generator in the pre-trained generative adversarial network is as follows: The standardized text is converted into a text feature vector, and the text feature vector is dimensionally aligned and fused with the context vector to obtain a joint feature; Based on the joint feature, a context-adapted frequency spectrum is generated by a context-aware generator; A pre-trained generative adversarial network is called to iteratively optimize the acoustic features and semantic consistency of the context-adapted frequency spectrum based on pre-training parameters; The optimized context-adapted frequency spectrum is converted into a time-domain waveform to generate an initial speech signal.

6. The method of claim 5, wherein the method is based on a generative adversarial network. The method for generating the context-adapted frequency spectrum by the context-aware generator based on the joint feature is as follows: A base mel-frequency spectrum is generated based on the text feature vector; Based on the gating attention mechanism, a frequency attention vector matching the dimension of the context vector is configured for each mel-frequency band, and the weight distribution of the frequency attention vector is dynamically adjusted according to the scene urgency; The joint feature and the adjusted frequency attention vector are associated to generate a weight coefficient of each mel-frequency band; The weight coefficient is used to perform frequency-band-by-frequency-band weighting optimization on the base mel-frequency spectrum to obtain the context-adapted frequency spectrum.

7. The method of claim 5, wherein the method is based on a generative adversarial network. In the process of generating the initial speech signal, an incremental update mechanism is adopted: The regeneration of the context vector is triggered by real-time monitoring of the original context feature change amount, and the context vector of the previous moment is used when the change amount does not reach a preset threshold. 8.The method of claim 1, wherein, The method for dynamic precision quantization is as follows: The target quantization precision level is determined according to the scene urgency, and the quantization precision level includes full-model low-precision quantization, mixed-precision quantization, and full-model high-precision quantization; The target quantization precision level is applied to the acoustic model, and the reference speech signal is converted by the acoustic model to generate the final speech.

9. An aviation speech generation system based on a generative adversarial network, characterized by, The system comprises: An information processing module configured to receive a text instruction and synchronously collect multi-dimensional aviation context information, perform ICAO standard terminology standardization processing on the text instruction, perform timestamp synchronization processing on the multi-dimensional aviation context information, and output a standardized text and an original context feature; A context generation module configured to calculate a scene urgency and encode a context vector based on the original context feature, and generate an initial speech signal by a pre-trained context-aware generator in combination with the context vector and the standardized text; wherein the process of generating the initial speech signal includes dynamic optimization of mel-frequency bands by a gating attention mechanism and real-time adaptation of the context vector by an incremental update mechanism; A collaborative verification module configured to perform collaborative verification on the initial speech signal based on a double-path ASR discriminator, and output a feedback signal to the context generation module when the verification fails; A speech generation module configured to perform scene-adaptive post-processing on the reference speech signal according to the scene urgency, generate a final speech, and transmit the final speech to an airborne broadcasting system for broadcasting; wherein the scene-adaptive post-processing includes multi-source noise separation, dynamic precision quantization, and loudness adaptation. The dual-path ASR discriminator includes a main-path ASR discriminator and an auxiliary-path ASR discriminator; the initial speech signal is cooperatively verified to obtain a reference speech signal, and the method is as follows: The initial speech signal is identified by using the main-path ASR discriminator to generate a first identification result; wherein the main-path ASR discriminator takes aviation special terms as an identification reference; The initial speech signal is identified by using the auxiliary-path ASR discriminator, deviation compensation is performed through feature mapping and regularization optimization, and a second identification result is generated; A word error rate is calculated based on the first identification result and the standardized text, and an adversarial calibration loss is calculated according to the difference between the first identification result and the second identification result; When the word error rate is zero and the adversarial calibration loss is lower than a preset threshold, it is determined that the verification is passed, and it is confirmed that the current speech signal is a reference speech signal; otherwise, the context-aware generator is fed back and optimized in combination with the word error rate and the adversarial calibration loss, and the initial speech signal is regenerated.

Images