Air traffic control fatigue monitoring system and method based on voiceprint recognition

Abstract

The invention discloses an air traffic control fatigue monitoring system and method based on voiceprint recognition, relates to the technical field of fatigue monitoring, and solves the problems that the fatigue state of a corresponding air controller is monitored by using a unified voiceprint reference in the existing fatigue monitoring technology, so that the fatigue state judgment accuracy of the air controller is reduced, and the fatigue monitoring efficiency is improved. And the efficiency of voiceprint monitoring of the fatigue state is reduced. The sound acquisition module is used for acquiring on-site sound data; the reference voiceprint construction unit is used for constructing reference voiceprint feature data according to the initial state voice; the semantic processing unit is used for generating a semantic association degree score according to the current state voice; the voiceprint processing unit is used for generating current voiceprint feature data according to the current state voice; generating a voiceprint state score according to the current voiceprint feature data and the reference voiceprint feature data; the alarm module is used for fusing the semantic association degree score and the voiceprint state score to obtain a fatigue degree score; and the fatigue state judgment accuracy is improved.

Description

Technical Field

[0001] This application belongs to the field of fatigue monitoring, and specifically relates to an air traffic control fatigue monitoring system and method based on voiceprint recognition. Background Art

[0002] With the rapid development of the aviation industry, the importance of air traffic control, that is, air traffic management, has become increasingly prominent. Air traffic controllers are responsible for ensuring the safe takeoff, landing, and route planning of aircraft, and their mental state directly affects aviation safety. Long working hours, irregular work schedules, and high-intensity work rhythms make air traffic controllers prone to fatigue. Fatigue not only reduces work efficiency but may also lead to serious safety accidents. In recent years, with the rapid development of speech recognition and voiceprint recognition technologies, new ideas have been provided for fatigue monitoring. Voiceprint recognition technology can identify an individual's identity and monitor the state of the voice by extracting and analyzing the unique characteristics of an individual's voice. Using voiceprint recognition technology to monitor the fatigue state of air traffic controllers has the advantages of being non-contact, highly real-time, and easy to operate, and is expected to become an important means to improve the level of aviation safety management.

[0003] Existing methods for monitoring fatigue states using voiceprints often involve establishing a personal voice database, generating voiceprint features of relevant personnel from the voice database, and then using the voiceprint features to determine whether relevant air traffic controllers are fatigued; however, air traffic controllers may have different working states every day due to physical discomfort, and the voiceprint features will also vary slightly; using a unified voiceprint benchmark to monitor the fatigue state of corresponding air traffic controllers will result in a decrease in the accuracy of judging the fatigue state of air traffic controllers, and in severe cases, more serious accidents may occur; this leads to a decrease in the efficiency of monitoring fatigue states using voiceprints. Therefore, an air traffic control fatigue monitoring system and method based on voiceprint recognition are needed. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems existing in the prior art; to this end, this application proposes an air traffic control fatigue monitoring system and method based on voiceprint recognition, which is used to solve the problem that using a unified voiceprint benchmark to monitor the fatigue state of corresponding air traffic controllers in existing fatigue monitoring technologies will result in a decrease in the accuracy of judging the fatigue state of air traffic controllers, and further lead to a decrease in the efficiency of monitoring fatigue states using voiceprints.

[0005] To achieve the above object, the first aspect of this application provides an air traffic control fatigue monitoring system based on voiceprint recognition, including: a sound acquisition module, a sound analysis module, an alarm module, and a database;

[0006] The sound acquisition module: is used to acquire on-site sound data;

[0007] The sound analysis module includes a reference voiceprint construction unit, a semantic processing unit, and a voiceprint processing unit;

[0008] Reference voiceprint construction unit: construct reference voiceprint feature data according to the initial state voice in the voice data;

[0009] Semantic processing unit: convert the current state voice in the voice data into a number of dialogue statements; input the number of dialogue statements into the relevance scoring model to obtain the semantic relevance score;

[0010] Voiceprint processing unit: generate current voiceprint feature data according to the current state voice in the voice data; generate a voiceprint state score according to the current voiceprint feature data and the reference voiceprint feature data;

[0011] Alarm module: generate a fatigue score according to the semantic relevance score and the voiceprint state score, and generate a fatigue warning signal according to the fatigue score.

[0012] This application obtains the initial state voice, adjusts the original reference voiceprint feature data in the database according to the initial state voice to construct the current reference voiceprint feature data; obtains the current state voice, and generates a semantic relevance score and a voiceprint state score according to the current state voice; obtains the current state voice, and generates a semantic relevance score and a voiceprint state score according to the current state voice; generates a fatigue warning signal according to the fatigue score; through correcting the reference voiceprint feature data, as well as performing semantic analysis on the voice data and performing voiceprint analysis on the voice data by correcting the reference voiceprint feature data, comprehensively analyzes the fatigue situation of air traffic controllers, increases the accuracy of fatigue state judgment; and further improves the efficiency of monitoring the fatigue state by voiceprint.

[0013] Preferably, constructing the reference voiceprint feature data according to the initial state voice in the voice data includes:

[0014] Preprocess the initial state voice to obtain its corresponding initial voiceprint feature data; the initial voiceprint feature data is the frequency domain feature obtained according to the initial state voice, including the amplitude values of each frequency band; obtain the original voiceprint feature data of the corresponding person through the connected database; the original voiceprint feature data is the frequency domain feature obtained by processing the normal historical initial state voice; includes the average value of the amplitude values and the amplitude value fluctuation variance of each frequency band; extract the amplitude value mean and amplitude value variance corresponding to each frequency band in the original voiceprint feature data;

[0015] Calculate the absolute value of the difference between the amplitude value of each frequency band in the initial voiceprint feature data and the average value of the amplitude values of the corresponding frequency bands in the original voiceprint feature data, and determine whether the absolute value of the difference is less than the variance of the amplitude values of the corresponding frequency bands in the original voiceprint feature data; if so, mark the frequency band status corresponding to the frequency band as a normal frequency band, and mark the amplitude value corresponding to the frequency band as the reference value; if not, mark the frequency band status corresponding to the frequency band as an abnormal frequency band, and mark the average value of the amplitude values corresponding to the frequency band as the reference value;

[0016] Generate reference voiceprint feature data according to the frequency band status and reference value corresponding to each frequency band.

[0017] Preferably, preprocess the voice in the initial state to obtain the corresponding initial voiceprint feature data, including:

[0018] Obtain the voice in the initial state, perform frame segmentation, windowing and denoising on the voice in the initial state, and then perform Fourier transform to obtain the corresponding spectrogram; obtain the amplitude values corresponding to each frequency band, and integrate the amplitude values corresponding to each frequency band into the initial voiceprint feature data.

[0019] Preferably, the generating of the reference voiceprint feature data according to the frequency band status and reference value corresponding to each frequency band includes:

[0020] Obtain the frequency band status corresponding to each frequency band, and determine whether the proportion of abnormal frequency band statuses is greater than the set abnormal proportion threshold; if so, generate a personnel status abnormal signal; if not, it means that the voice in the initial state is normal, and integrate the reference values corresponding to each frequency band into the reference voiceprint feature data.

[0021] Preferably, the relevance scoring model is trained in the following manner:

[0022] Obtain a number of dialogue sentences and their corresponding semantic relevance scores through the database; the semantic relevance score is the score given by an expert to the degree of relevance of the sentences from different personnel in the dialogue sentence, and the greater the degree of relevance of the personnel dialogue in the dialogue sentence, the greater the corresponding semantic relevance score; integrate the semantic relevance score and its corresponding dialogue sentence into a number of training data and test data;

[0023] Use the training data to train the artificial intelligence model, and use the test data to test the trained artificial intelligence model, and finally obtain a relevance scoring model with the input being the dialogue sentence and the output being the corresponding semantic relevance score; among them, the artificial intelligence model includes a BP neural network model and an RBF neural network model.

[0024] Preferably, the generating of the current voiceprint feature data according to the current state voice in the voice data includes:

[0025] Extract the voices of air traffic controllers in each dialogue statement in the voice data;

[0026] Preferably, the generation of the voiceprint status score based on the current voiceprint feature data and the reference voiceprint feature data includes:

[0027] Obtain the amplitude values corresponding to each frequency band in the current voiceprint feature data and the reference values of each frequency band in the reference voiceprint feature data; calculate the voiceprint status score of the corresponding frequency band according to the ratio of the amplitude value corresponding to each frequency band to the reference value; calculate the voiceprint status scores corresponding to each frequency band in turn.

[0028] Preferably, the generation of the fatigue score based on the semantic correlation score and the voiceprint status score includes:

[0029] Obtain the reference values corresponding to each frequency band in the reference voiceprint feature, the average amplitude value corresponding to each frequency band in the original voiceprint feature data, and the set fatigue ratio threshold; the fatigue ratio threshold is the ratio of the amplitude value in the fatigue state to the average amplitude value for each frequency band; update the fatigue ratio threshold according to the reference value and the average amplitude value of each frequency band;

[0030] Obtain the semantic correlation score corresponding to each dialogue statement and the voiceprint status scores of its corresponding frequency bands;

[0031] Calculate the sentence fatigue score of the dialogue statement according to the semantic correlation score corresponding to the dialogue statement and the voiceprint status scores of its corresponding frequency bands;

[0032] Obtain the sentence fatigue scores of each dialogue statement, and perform weighted summation on them to obtain the fatigue score for this time; each weight coefficient in the weighted summation is set according to experience. In this embodiment, time is used for setting. The closer the voice acquisition time corresponding to the dialogue statement is to the current time, the larger the corresponding weight coefficient is set.

[0033] Preferably, updating the fatigue ratio threshold according to the reference value and the average amplitude value of each frequency band includes:

[0034] When the reference value corresponding to the frequency band is greater than its corresponding average amplitude value, add a set proportion of the value to its original fatigue ratio threshold to obtain the updated fatigue ratio threshold corresponding to the frequency band;

[0035] When the reference value corresponding to the frequency band is less than its corresponding average amplitude value, add a set proportion of the value to its original fatigue ratio threshold to obtain the updated fatigue ratio threshold corresponding to the frequency band;

[0036] Preferably, the formula for calculating the sentence fatigue score based on the semantic correlation score corresponding to the dialogue sentence and the voiceprint state scores of its corresponding frequency bands is as follows:

[0037]

[0038] YJPj is the sentence fatigue score of the j-th dialogue sentence; YPj is the semantic correlation score of the j-th dialogue sentence; SWPij is the voiceprint state score corresponding to the i-th frequency band in the j-th dialogue sentence; PLYi is the fatigue ratio threshold corresponding to the i-th frequency band; α1 and α2 are weight coefficients for adjusting the influence degrees of the semantic correlation score and the voiceprint state score on the sentence fatigue score, and the specific values are set according to expert experience; β1 and β2 are proportionality coefficients for adjusting the influence of different voiceprint state scores on the sentence fatigue score, and the specific values are set according to expert experience, and β1 < β2;

[0039] Preferably, generating a fatigue warning signal according to the fatigue score includes:

[0040] Obtain the fatigue score, and when the fatigue score is greater than the set score threshold, generate a fatigue warning signal.

[0041] Another aspect of the present application provides an air traffic control fatigue monitoring method based on voiceprint recognition, including the following steps:

[0042] Step 1: Obtain the initial state voice, and construct the reference voiceprint feature data according to the initial state voice;

[0043] Step 2: Obtain the current state voice, and generate a semantic correlation score and a voiceprint state score according to the current state voice;

[0044] Step 3: Generate a fatigue score according to the semantic correlation score and the voiceprint state score;

[0045] Step 4: Generate a fatigue warning signal according to the fatigue score.

[0046] Preferably, step 2 includes the following steps:

[0047] S21: Obtain the current state voice;

[0048] S22: Convert the current state voice in the sound data into a number of dialogue sentences; input the number of dialogue sentences into the correlation score model to obtain the semantic correlation score;

[0049] S23: Generate the current voiceprint feature data according to the current state voice in the sound data; generate the voiceprint state score according to the current voiceprint feature data and the reference voiceprint feature data.

[0050] Compared with the prior art, the beneficial effects of the present application are as follows:

[0051] 1. The present application obtains the initial state voice, adjusts and constructs the current reference voiceprint feature data for the original reference voiceprint feature data in the database according to the initial state voice; obtains the current state voice, and generates a semantic correlation score and a voiceprint state score according to the current state voice; obtains the current state voice, and generates a semantic correlation score and a voiceprint state score according to the current state voice; generates a fatigue warning signal according to the fatigue score; by correcting the reference voiceprint feature data, and through the voiceprint analysis of the voice data by correcting the reference voiceprint feature data, the accuracy of fatigue state judgment is increased; thereby improving the efficiency of voiceprint monitoring of the fatigue state.

[0052] 2. The present application comprehensively analyzes the fatigue situation of air traffic controllers by performing semantic analysis and voiceprint analysis on voice data, further increasing the accuracy of fatigue state judgment; improving the efficiency of voiceprint monitoring of the fatigue state. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0054] Figure 1 It is a schematic diagram of the module connection of the fatigue monitoring system in the present application;

[0055] Figure 2 It is a schematic flow diagram of the fatigue monitoring method in the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The following will clearly and completely describe the technical solutions of the present application in conjunction with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0057] Please refer to Figure 1 , the first aspect of the present application provides an air traffic control fatigue monitoring system based on voiceprint recognition, including: a sound collection module, a sound analysis module, an alarm module, and a database;

[0058] Sound collection module: used to collect on-site sound data; the sound data is the voice data of the air traffic controller collected, including the initial state voice recorded during work, and the current state voice, that is, the current conversation voice;

[0059] The voice analysis module includes a reference voiceprint construction unit, a semantic processing unit, and a voiceprint processing unit;

[0060] Reference voiceprint construction unit: Construct reference voiceprint feature data based on the initial state voice in the voice data; the reference voiceprint feature data is the voiceprint feature of normal speech set according to the state of the air traffic controller when going to work on the current day;

[0061] Semantic processing unit: Convert the current state voice in the voice data into several dialogue statements, that is, convert the voice into text. The dialogue statements include the content of the air traffic controller's voice and the content of the voices of other relevant personnel; input the several dialogue statements into the correlation scoring model to obtain the semantic correlation score; the voice correlation score is the relevance of the dialogue content between the air traffic controller and other relevant personnel. The greater the degree of relevance, the greater the corresponding voice correlation score;

[0062] Voiceprint processing unit: Generate current voiceprint feature data based on the current state voice in the voice data; the current voiceprint feature data is the relevant voice feature generated according to the current voice of the air traffic controller; generate a voiceprint state score based on the current voiceprint feature data and the reference voiceprint feature data. The voiceprint state score is the result of comparing the current voiceprint feature data with the reference voiceprint feature data, that is, it represents the current voice state of the air traffic controller. The closer the voice state is to the initial voice state, the higher the corresponding voiceprint state score;

[0063] Alarm module: Generate a fatigue score based on the semantic correlation score and the voiceprint state score. The fatigue score is a score used to judge the fatigue degree of the air traffic controller obtained through comprehensive analysis of the semantic correlation score and the voiceprint state score; the higher the fatigue score, the more serious the fatigue situation of the air traffic controller; generate a fatigue warning signal based on the fatigue score.

[0064] In this embodiment, by obtaining the initial state voice, adjusting the original reference voiceprint feature data in the database according to the initial state voice to construct the current reference voiceprint feature data; obtaining the current state voice, generating a semantic correlation score and a voiceprint state score according to the current state voice; obtaining the current state voice, generating a semantic correlation score and a voiceprint state score according to the current state voice; generating a fatigue warning signal according to the fatigue score; through correcting the reference voiceprint feature data, as well as performing semantic analysis on the voice data and performing voiceprint analysis on the voice data by correcting the reference voiceprint feature data, comprehensively analyzing the fatigue situation of the air traffic controller, increasing the accuracy of fatigue state judgment; thereby improving the efficiency of voiceprint monitoring of the fatigue state.

[0065] Constructing the reference voiceprint feature data according to the initial state voice in the voice data includes:

[0066] Preprocess the initial state voice to obtain its corresponding initial voiceprint feature data; the initial voiceprint feature data is the frequency domain feature obtained from the initial state voice, including the amplitude values of each frequency band; obtain the original voiceprint feature data of the corresponding person through the connected database; the original voiceprint feature data is the frequency domain feature obtained by processing the normal historical initial state voice; it includes the average value of the amplitude values and the variance of the amplitude value fluctuations of each frequency band; extract the mean value of the amplitude values and the variance of the amplitude values corresponding to each frequency band in the original voiceprint feature data; specifically, the mean value of the amplitude values and the variance of the amplitude values are obtained through the following methods: obtain a number of normal historical state voice data; perform frame segmentation, windowing, and denoising on the historical state voice and then perform Fourier transform to obtain the corresponding spectrogram; obtain the mean value of a number of amplitude values corresponding to a certain frequency band as the mean value of the amplitude values of the corresponding frequency band; take the variance of a number of amplitude values corresponding to a certain frequency band as the variance of the amplitude values of the corresponding frequency band;

[0067] It can be understood that the initial state voice is the set voice input by the air traffic controller before officially starting work, and the content of this voice includes the characteristics of each frequency band; the historical initial state voice is the initial state voice that the corresponding air traffic controller has successfully passed in the past;

[0068] Calculate the absolute value of the difference between the amplitude value of each frequency band in the initial voiceprint feature data and the mean value of the amplitude values of the corresponding frequency band in the original voiceprint feature data, and determine whether the absolute value of the difference is less than the variance of the amplitude values of the corresponding frequency band in the original voiceprint feature data; if so, mark the frequency band state corresponding to the frequency band as a normal frequency band, and mark the amplitude value corresponding to the frequency band as the reference value; if not, mark the frequency band state corresponding to the frequency band as an abnormal frequency band, and mark the mean value of the amplitude values corresponding to the frequency band as the reference value;

[0069] Generate the reference voiceprint feature data according to the frequency band state and the reference value corresponding to each frequency band.

[0070] It can be understood that in this embodiment, only the amplitude value in the sound feature in the frequency domain is taken as an example. Combining other voiceprint features will make the final fatigue monitoring result more accurate;

[0071] Preprocess the initial state voice to obtain its corresponding initial voiceprint feature data, including: obtain the initial state voice, perform frame segmentation, windowing, and denoising on the initial state voice and then perform Fourier transform to obtain the corresponding spectrogram; obtain the amplitude values corresponding to each frequency band, and integrate the amplitude values corresponding to each frequency band into the initial voiceprint feature data. It can be understood that the above frequency band is the frequency band divided from the frequency in the spectrogram according to the set step size, and the length of each frequency band is the same as the set step size.

[0072] Generate reference voiceprint feature data based on the frequency band status and reference values corresponding to each frequency band, including: obtaining the frequency band status corresponding to each frequency band, and determining whether the proportion of abnormal frequency band statuses is greater than a set abnormal proportion threshold; if so, generate a personnel status abnormal signal; otherwise, it indicates that the initial state voice is normal, and integrate the reference values corresponding to each frequency band into reference voiceprint feature data.

[0073] The correlation score model is trained in the following way: obtain a number of dialogue sentences and their corresponding semantic correlation scores through a database; the semantic correlation score is the score given by an expert on the degree of relevance of sentences from different personnel in the dialogue sentence, and the greater the degree of relevance of the personnel dialogue within the dialogue sentence, the greater the corresponding semantic correlation score; integrate the semantic correlation scores and their corresponding dialogue sentences into a number of training data and test data;

[0074] Use the training data to train the artificial intelligence model, and use the test data to test the trained artificial intelligence model, and finally obtain a correlation score model with the input being the dialogue sentence and the output being the corresponding semantic correlation score; among them, the artificial intelligence model includes a BP neural network model and an RBF neural network model; the specific steps of training the artificial intelligence are existing technologies and will not be elaborated here too much.

[0075] The specific steps of generating the current voiceprint feature data from the current state voice in the voice data and preprocessing the initial state voice to obtain its corresponding initial voiceprint feature data are similar; specifically, perform frame segmentation, windowing, and denoising on the current state voice and then perform Fourier transform to obtain the corresponding spectrogram; obtain the amplitude values corresponding to each frequency band, and integrate the amplitude values corresponding to each frequency band into the current voiceprint feature data; it can be understood that the frequency bands in the current voiceprint feature data correspond one-to-one and are the same as those in the initial voiceprint feature data.

[0076] Generate a voiceprint status score based on the current voiceprint feature data and the reference voiceprint feature data, including: obtaining the amplitude values corresponding to each frequency band in the current voiceprint feature data, and the reference values corresponding to each frequency band in the reference voiceprint feature data; calculate the voiceprint status score corresponding to the corresponding frequency band according to the ratio of the amplitude value corresponding to each frequency band to the reference value; calculate the voiceprint status scores corresponding to each frequency band in turn; specifically, through the formula:

[0077]

[0078] Calculate the voiceprint status score SWPi corresponding to the frequency band numbered i; where, ZFi is the amplitude value corresponding to the frequency band numbered i; ZFJi is the reference value corresponding to the frequency band numbered i; i is the number of the frequency band.

[0079] Generate a fatigue score based on the semantic relevance score and the voiceprint status score, including: obtaining the reference value corresponding to each frequency band in the reference voiceprint feature, as well as the mean amplitude value corresponding to each frequency band in the original voiceprint feature data and the set fatigue ratio threshold; the fatigue ratio threshold is the ratio of the amplitude value in the fatigue state to the mean amplitude value under each frequency band; update the fatigue ratio threshold according to the reference value and the mean amplitude value of each frequency band;

[0080] Obtain the semantic relevance score corresponding to each dialogue statement and the voiceprint status score of each frequency band corresponding thereto; calculate the statement fatigue score of the dialogue statement according to the semantic relevance score corresponding to the dialogue statement and the voiceprint status score of each frequency band corresponding thereto;

[0081] Obtain the statement fatigue scores of each dialogue statement, and perform weighted summation on them to obtain the fatigue score for this time; in the weighted summation, each weight coefficient is set according to experience. In this embodiment, time is used for setting. The closer the voice acquisition time corresponding to the dialogue statement is to the current time, the larger the corresponding weight coefficient is set.

[0082] Update the fatigue ratio threshold according to the reference value and the mean amplitude value of each frequency band, including: when the reference value corresponding to the frequency band is greater than its corresponding mean amplitude value, add a set proportion of the value to its original fatigue ratio threshold to obtain the updated fatigue ratio threshold corresponding to the frequency band; specifically, obtain the reference value SFJi, the mean amplitude value ZFJi, and the original fatigue ratio threshold YPLYi corresponding to the frequency band numbered i; through the formula Calculate the updated fatigue ratio threshold corresponding to the frequency band numbered i, where ZFD is the amplitude value unit coefficient used to remove the units of the reference value and the mean amplitude value; KLY is the set adjustable threshold range;

[0083] When the reference value corresponding to the frequency band is less than its corresponding mean amplitude value, add a set proportion of the value to its original fatigue ratio threshold to obtain the updated fatigue ratio threshold corresponding to the frequency band; specifically, obtain the reference value SFJi, the mean amplitude value ZFJi, and the original fatigue ratio threshold YPLYi corresponding to the frequency band numbered i; through the formula Calculate the updated fatigue ratio threshold corresponding to the frequency band numbered i, where ZFD is the amplitude value unit coefficient used to remove the units of the reference value and the mean amplitude value; KLY is the set adjustable threshold range; ε1 and ε2 are the set proportionality coefficients used to adjust the influence of the excited state on the fatigue ratio threshold and the influence of the micro-fatigue state on the fatigue ratio threshold.

[0084] In this embodiment, the updated fatigue ratio threshold is calculated through the above two formulas; the greater the difference between the reference value and the average amplitude value, the greater the difference between the current state of the air traffic controller and the normal state. Whether it is excitement or mild fatigue, it will cause the air traffic controller to consume energy too quickly and reach the fatigue state in advance. To ensure the early detection of the fatigue state of the air traffic controller, this embodiment appropriately increases the fatigue ratio threshold.

[0085] The formula for calculating the sentence fatigue score based on the semantic correlation score corresponding to the dialogue sentence and the voiceprint state scores of its corresponding frequency bands is as follows:

[0086]

[0087] YJPj is the sentence fatigue score of the jth dialogue sentence; YPj is the semantic correlation score of the jth dialogue sentence; SWPij is the voiceprint state score corresponding to the frequency band numbered i in the jth dialogue sentence; PLYi is the fatigue ratio threshold corresponding to the frequency band numbered i; α1 and α2 are weight coefficients that adjust the influence degrees of the semantic correlation score and the voiceprint state score on the sentence fatigue score, and the specific values are set according to expert experience; β1 and β2 are proportionality coefficients used to adjust the influence of different voiceprint state scores on the sentence fatigue score, and the specific values are set according to expert experience, and β1 < β2. In this embodiment, the semantic correlation score and the voiceprint state scores corresponding to each frequency band are fused into the sentence fatigue score through the above formula; when the semantic correlation of the corresponding dialogue sentence is stronger, it indicates that the dialogue between the air traffic controller and the relevant personnel on the plane is smoother and there are no problems with the content of the dialogue. Furthermore, it indicates that the current state of the relevant air traffic controller is relatively sober, and the corresponding sentence fatigue score is set relatively small; when the voiceprint state scores corresponding to each frequency band of the air traffic controller's voice in the dialogue sentence are all less than the corresponding fatigue ratio threshold, it indicates that from the perspective of voiceprint characteristics, the air traffic controller is in a fatigue state, and the more it is less than, the more serious the fatigue state; the corresponding sentence fatigue score is also set relatively large. This embodiment comprehensively considers the dialogue semantics and voiceprint characteristics during the dialogue process of the air traffic controller to judge whether the air traffic controller is fatigued, making the judgment result more accurate.

[0088] Generate a fatigue warning signal based on the fatigue score, including: obtaining the fatigue score, and generating a fatigue warning signal when the fatigue score is greater than the set score threshold.

[0089] Please refer to Figure 2 , another aspect of this application provides an air traffic control fatigue monitoring method based on voiceprint recognition, including the following steps:

[0090] Step 1: Obtain the initial state voice and construct the reference voiceprint feature data according to the initial state voice;

[0091] Step 2: Obtain the current state voice, and generate a semantic correlation score and a voiceprint state score based on the current state voice;

[0092] Step 3: Generate a fatigue score based on the semantic correlation score and the voiceprint state score;

[0093] Step 4: Generate a fatigue warning signal based on the fatigue score.

[0094] Preferably, the said Step 2 includes the following steps:

[0095] S21: Obtain the current state voice;

[0096] S22: Convert the current state voice in the voice data into a number of dialogue sentences; input the number of dialogue sentences into the correlation score model to obtain the semantic correlation score;

[0097] S23: Generate the current voiceprint feature data based on the current state voice in the voice data; generate the voiceprint state score based on the current voiceprint feature data and the reference voiceprint feature data.

[0098] Some of the data in the above formula is the numerical value calculated after removing the dimension. The formula is obtained by software simulation of a large amount of collected data to get a formula closest to the actual situation; the preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0099] The working principle of this application:

[0100] This application obtains the initial state voice, adjusts the original reference voiceprint feature data in the database according to the initial state voice to construct the current reference voiceprint feature data; obtains the current state voice, and generates a semantic correlation score and a voiceprint state score based on the current state voice; obtains the current state voice, and generates a semantic correlation score and a voiceprint state score based on the current state voice; generates a fatigue warning signal based on the fatigue score; through correcting the reference voiceprint feature data, as well as performing semantic analysis on the voice data and performing voiceprint analysis on the voice data by correcting the reference voiceprint feature data, comprehensively analyzes the fatigue situation of air traffic controllers, increases the accuracy of fatigue state judgment; and further improves the efficiency of monitoring fatigue state by voiceprint.

[0101] The above embodiments are only used to illustrate the technical method of this application and not to limit it. Although this application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical method of this application can be modified or equivalently replaced without departing from the spirit and scope of the technical method of this application.

Claims

1. An air traffic control fatigue monitoring system based on voiceprint recognition, comprising: A sound acquisition module, a sound analysis module, an alarm module, and a database. The sound analysis module includes a reference voiceprint construction unit, a semantic processing unit, and a voiceprint processing unit. It is characterized in that The sound acquisition module: is used to acquire the sound data on-site; The reference voiceprint construction unit: constructs reference voiceprint feature data according to the initial state voice in the sound data; The semantic processing unit: converts the current state voice in the sound data into a number of dialogue statements; inputs the number of dialogue statements into the relevance scoring model to obtain the semantic relevance score; The voiceprint processing unit: performs Fourier transform on the current state voice in the sound data to obtain the current voiceprint feature data; compares the current voiceprint feature data with the reference voiceprint feature data to obtain the voiceprint state score; The alarm module: fuses the semantic relevance score and the voiceprint state score to obtain the fatigue score, and generates a fatigue warning signal according to the fatigue score.

2. The air traffic control fatigue monitoring system based on voiceprint recognition according to claim 1, characterized in that Constructing the reference voiceprint feature data according to the initial state voice in the sound data includes: Performing preprocessing on the initial state voice to obtain its corresponding initial voiceprint feature data; the initial voiceprint feature data is the frequency domain feature obtained according to the initial state voice, including the amplitude values of each frequency band; obtaining the original voiceprint feature data of the corresponding person through the connected database; the original voiceprint feature data is the frequency domain feature obtained by processing the normal historical initial state voice; including the average value of the amplitude values of each frequency band and the amplitude value fluctuation variance; extracting the amplitude value mean and amplitude value variance corresponding to each frequency band in the original voiceprint feature data; Calculating the absolute value of the difference between the amplitude value of each frequency band in the initial voiceprint feature data and the amplitude value mean of the corresponding frequency band in the original voiceprint feature data, and judging whether the absolute value of the difference is less than the amplitude value variance of the corresponding frequency band in the original voiceprint feature data; if so, mark the frequency band state corresponding to the frequency band as a normal frequency band, and mark the amplitude value corresponding to the frequency band as the reference value; if not, mark the frequency band state corresponding to the frequency band as an abnormal frequency band, and mark the amplitude value mean of the corresponding frequency band as the reference value; Generating reference voiceprint feature data according to the frequency band state and reference value corresponding to each frequency band.

3. The air traffic control fatigue monitoring system based on voiceprint recognition according to claim 2, wherein, Performing preprocessing on the initial state voice to obtain its corresponding initial voiceprint feature data includes: Obtaining the initial state voice, performing frame segmentation, windowing, and denoising on the initial state voice, and then performing Fourier transform to obtain the corresponding spectrogram; obtaining the amplitude values corresponding to each frequency band, and integrating the amplitude values corresponding to each frequency band into the initial voiceprint feature data.

4. The air traffic control fatigue monitoring system based on voiceprint recognition according to claim 2, characterized in that, The generating reference voiceprint feature data according to the frequency band state and reference value corresponding to each frequency band includes: Obtaining the frequency band state corresponding to each frequency band, and judging whether the proportion of the abnormal frequency band state is greater than the set abnormal proportion threshold; if so, generating a person state abnormal signal; if not, integrating the reference values corresponding to each frequency band into the reference voiceprint feature data.

5. The air traffic control fatigue monitoring system based on voiceprint recognition according to claim 1, characterized in that, The relevance scoring model is trained in the following way: Obtain a number of dialogue statements and their corresponding semantic correlation scores from a database; the semantic correlation score is the score given by an expert for the degree of relevance of statements from different speakers in the dialogue statement; Integrate the semantic correlation scores and their corresponding dialogue statements into a number of training data and test data; Use the training data to train an artificial intelligence model, and use the test data to test the trained artificial intelligence model. Finally, obtain a correlation score model with dialogue statements as input and corresponding semantic correlation scores as output.

6. The air traffic control fatigue monitoring system based on voiceprint recognition according to claim 1, characterized in that, The generating of the voiceprint state score according to the current voiceprint feature data and the reference voiceprint feature data includes: Obtain the amplitude values corresponding to each frequency band in the current voiceprint feature data and the reference values of each frequency band in the reference voiceprint feature data; calculate the voiceprint state score of the corresponding frequency band according to the ratio of the amplitude value corresponding to each frequency band to the reference value; calculate the voiceprint state scores corresponding to each frequency band in turn.

7. The air traffic control fatigue monitoring system based on voiceprint recognition according to claim 1, wherein, The generating of the fatigue score according to the semantic correlation score and the voiceprint state score includes: Obtain the reference values corresponding to each frequency band in the reference voiceprint feature, the average amplitude value corresponding to each frequency band in the original voiceprint feature data, and a set fatigue ratio threshold; the fatigue ratio threshold is the ratio of the amplitude value in the fatigue state to the average amplitude value for each frequency band; update the fatigue ratio threshold according to the reference value and the average amplitude value of each frequency band; Obtain the semantic correlation scores corresponding to each dialogue statement and the voiceprint state scores of each frequency band corresponding thereto; Calculate the statement fatigue score of the dialogue statement according to the semantic correlation score corresponding to the dialogue statement and the voiceprint state scores of each frequency band corresponding thereto; Obtain the statement fatigue scores of each dialogue statement, and perform weighted summation on them to obtain the fatigue score for this time.

8. The air traffic control fatigue monitoring system based on voiceprint recognition according to claim 7, characterized in that, The updating of the fatigue ratio threshold according to the reference value and the average amplitude value of each frequency band includes: When the reference value corresponding to the frequency band is greater than its corresponding average amplitude value, add a set proportion of the value to its original fatigue ratio threshold to obtain the updated fatigue ratio threshold corresponding to the frequency band; When the reference value corresponding to the frequency band is less than its corresponding average amplitude value, add a set proportion of the value to its original fatigue ratio threshold to obtain the updated fatigue ratio threshold corresponding to the frequency band.

9. A method for monitoring air traffic control fatigue based on voiceprint recognition, which is based on the operation of a system for monitoring air traffic control fatigue based on voiceprint recognition according to any one of claims 1 to 8; characterized in that, It includes the following steps: Step 1: Obtain the initial state voice, and construct the reference voiceprint feature data according to the initial state voice; Step 2: Obtain the current state voice, and generate the semantic correlation score and the voiceprint state score according to the current state voice; Step 3: Generate the fatigue score according to the semantic correlation score and the voiceprint state score; Step 4: Generate a fatigue warning signal according to the fatigue score.

10. The method for monitoring air traffic control fatigue based on voiceprint recognition according to claim 9, characterized in that, The said Step 2 includes the following steps: S21: Obtain the current state voice; S22: Convert the current state voice in the sound data into a number of dialogue statements; input the number of dialogue statements into the correlation score model to obtain the semantic correlation score; S23: Generate the current voiceprint feature data according to the current state voice in the sound data; generate the voiceprint state score according to the current voiceprint feature data and the reference voiceprint feature data.

Images