Helium voice correction method based on generative adversarial network

Abstract

The invention provides a helium voice correction method based on a generative adversarial network, and belongs to the technical field of voice processing and artificial intelligence. The technical problem of helium voice distortion caused by pitch period distortion and formant offset in a helium-oxygen environment is solved. According to the technical scheme, the method comprises the following steps of S1, preprocessing helium voice data; s2, constructing a special helium voice conversion model; and S3, inputting the preprocessed helium voice data into a voice recognition and semantic similarity evaluation system for processing. The beneficial effects of the invention are that the GAN is adopted as an advanced deep learning model, and the GAN can effectively capture the spectrum details in the voice signals through the multi-layer structure and the excellent unsupervised learning mechanism, thereby remarkably improving the voice correction performance.

Description

Technical Field

[0001] The present invention relates to the field of speech processing and artificial intelligence technology, and in particular to a helium speech correction method based on a generative adversarial network. Background Art

[0002] As the global resource shortage problem becomes increasingly serious, the development and utilization of marine resources has become increasingly important. The ocean is seen as an important energy supply in the future and is expected to provide more than 50% of energy demand. This concept has greatly promoted the development of deep-sea activities and related technologies around the world, such as deep-sea diving, deep-sea mining, and deep-sea surveys. It also promotes the application of deep-sea saturation diving technology in various fields, such as marine development, navigation operations, and maritime rescue.

[0003] Working in deep-sea environments requires highly specialized technology. In addition to relying on advanced manned submersibles or underwater vehicles, saturation diving technology can also be used. Saturation diving is a special diving technology that enables divers to work for a long time in deep-sea environments. In saturation diving, divers will inhale a mixture of helium and oxygen. Only in this way can divers adapt to the high-pressure environment of the deep sea without experiencing the "nitrogen narcosis" effect. However, this helium-oxygen mixture will cause significant changes in the divers' voices, resulting in the "helium voice" phenomenon. The problems caused by helium voice distortion not only affect the judgment of the land support personnel on the divers' physiological state, but also affect the cooperation between divers, resulting in low safety and efficiency of diving operations. Therefore, solving helium voice distortion is an urgent problem to be solved. In order to meet this challenge, researchers have made various attempts, mainly using speech processing technology to improve the intelligibility of helium voice. Although traditional methods such as time domain processing, linear prediction and homomorphic signal processing technology have improved the intelligibility of helium voice to a certain extent, the corrected voice quality still does not meet the ideal standard. These methods mainly focus on solving the problem of formant frequency distortion, but are insufficient in dealing with noise influence and pitch correction.

[0004] At present, in helium speech processing, time domain segmentation processing is the most commonly used method for helium speech correction. The helium speech signal is divided into several segments, and then each segment is stretched and processed, and then the segments are spliced ​​into a corrected speech signal. Although this method can solve the problem of formant frequency distortion to a certain extent, it does not solve the problem of spectrum envelope correction. In order to achieve more complex spectrum envelope correction, Suzuki et al. proposed a linear prediction and homomorphic signal processing method for estimating and modifying the impulse response of the vocal tract, and used inter-segment autocorrelation technology to reduce the impact of noise on helium speech. Richards proposed a helium speech correction method based on short-time Fourier transform of speech signals. This method is similar to the time domain segmentation processing method. The difference is that the Fourier transform is performed on the frequency domain signal segment instead of processing the signal segment in the time domain. Although this method can map any spectral envelope, it cannot modify the pitch distortion. Subsequently, Duncan proposed a residual excitation linear prediction encoder, which improved the intelligibility and naturalness of speech output to a certain extent, but it is still far from practical application. There are other methods for correcting helium speech, such as improved speech-excited vocoders, analytical signal methods, and high-voltage speech transcoders.

[0005] At present, there are the following problems that need to be solved in helium speech correction: First, there is no publicly available data set for helium speech correction. Second, the processing method of helium speech still remains in the scope of using time domain and frequency domain processing. Then, traditional correction methods must meet certain conditions, and most correction functions are only used for a certain concentrated frequency and are not applicable to all frequencies. Finally, the correction of helium speech involves sentences and isolated words, and there is no effective evaluation indicator for sentences. Summary of the invention

[0006] The purpose of the present invention is to provide a helium speech correction method based on a generative adversarial network. The present invention adopts a generative adversarial network (GAN) as an advanced deep learning model. GAN can effectively capture the spectral details in the speech signal through its multi-layer structure and excellent unsupervised learning mechanism, and significantly improve the performance of speech correction.

[0007] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a helium speech correction method based on speech recognition and semantic similarity, comprising the following steps:

[0008] S1. Preprocessing the helium voice data, including denoising, pre-emphasis and other steps, to reduce the reduction of the voice spectrum amplitude in the high frequency range and ensure the clarity of the voice signal;

[0009] S2. Build a dedicated helium speech conversion model, which combines deep learning methods to convert the processed helium speech signal into standard speech. The model consists of two main parts: the generator and discriminator of the model. The speech feature extraction module extracts the features of the pre-processed speech, including the unique spectral features of helium speech; the speech conversion module uses the adaptive-audio based metric generative adversarial network (AMGAN) based on GAN to map the extracted features into normal speech signals, thereby improving the naturalness and clarity of the speech;

[0010] S3. The pre-processed Helium speech data is input into the speech recognition and semantic similarity evaluation system for processing, and two types of results are finally generated: one is the text output from the speech recognition module, and the other is the score based on semantic similarity. These two types of results together constitute the basis for evaluating the quality of Helium speech correction.

[0011] Furthermore, the step S1 includes the following steps:

[0012] S11, pre-emphasize the input audio sequence, pass the voice signal through a high-pass filter, so as to enhance the high-frequency part of the voice signal, and keep the signal in the entire frequency band from low frequency to high frequency, and after the pre-emphasis processing, the spectrum can be obtained using the same signal-to-noise ratio;

[0013] S12, for the pre-emphasized nodes, divide the audio signal into frames of fixed length, and use the frame length as the minimum unit;

[0014] S13, windowing is performed on the framed data, and adjacent frames need to have overlapping parts, that is, the distance the window moves each time is less than the frame length;

[0015] Furthermore, the step S2 includes the following steps:

[0016] S21. The framework is based on the adaptive speech generative adversarial network (AMGAN). AMGAN completes the learning of multiple input features, including FBank features and formant features. AMGAN learns different features and models the audio.

[0017] S22. Build an adaptive segmentation algorithm for AMGAN. The two parts of the adaptive segmentation algorithm can meet the differentiated processing requirements of speech with different frequencies.

[0018] S23. Construct the fusion loss function of the AMGAN framework. The fusion loss function is to better quantify the difference between the two speech segments, so that it can effectively convert the helium speech into normal speech.

[0019] Furthermore, in the step S21, constructing multiple input features of AMGAN includes the following steps:

[0020] S211, extract FBank features from the preprocessed audio to input into the model;

[0021] S212. After preprocessing the audio, an electrical transmission simulation model is established, the human vocal tract is approximated as a first-order lossy sound tube, and the resonance peak characteristics are extracted for input into the model.

[0022] Furthermore, in the step S211, the input features include the following steps:

[0023] S2111, performing DFT processing on the input data and then performing logarithm processing;

[0024] S2112, performing Mel filtering on the data processed by DFT;

[0025] Furthermore, in the step S212, the input features include the following steps:

[0026] S2121, establish an electrical transmission simulation model with the input data,

[0027] S2122, extracting resonance peaks from the electrical transmission simulation model;

[0028] Furthermore, in the step S2111, the data processing includes the following steps:

[0029] S21111, the windowed data is processed by DFT to extract a fixed number of features from each frame, and the frame signal is processed by discrete Fourier transform to obtain the power spectrum estimation of the periodogram. The formula is as follows

[0030]

[0031] Where S(n) represents the time domain signal, S i (n) represents the data of the i-th frame, n is the value range, S i (k) represents the kth complex coefficient of the i-th frame, P i (k) represents the power spectrum of the i-th frame, h(n) is the window function of N points, and k is the length of DFT;

[0032] Furthermore, in the step S2112, the data processing includes the following steps:

[0033] S21121. Perform Mel filtering on the data after DFT processing. After DFT, the amplitude of each frequency component is obtained. The energy of each frequency component is obtained by squaring the amplitude. Since the human ear has different sensitivities to different frequencies, it is more sensitive to low frequencies and less sensitive to high frequencies. In order to simulate this characteristic of the human ear, the frequency is mapped to the Mel frequency. The relationship between the Mel frequency and the frequency is as follows:

[0034]

[0035] Where k is the scaling factor that converts linear frequency into Mel frequency, and b is the reference point that reflects the frequency resolution characteristics of the auditory system.

[0036] Applying the Mel filter bank on the energy spectrum obtains the FBank feature, the formula is:

[0037]

[0038] Where Y is the corresponding feature;

[0039] Furthermore, in the step S2121, the data processing includes the following steps:

[0040] S21211. If the shape of the source spectrum remains unchanged, the amplitude of the helium speech spectrum in the high frequency range will be significantly reduced. Therefore, pre-emphasis processing must be performed before extraction:

[0041] s′(n)=s(n)-μs(n-1) (5)

[0042] Among them, μ is the pre-emphasis weight, and s represents the original signal.

[0043] Next is the prediction error filtering process, the filter is as follows:

[0044]

[0045] Where p is the order of the prediction error filter, and γi is the filter coefficient of each order. The complex roots of the polynomial represent the resonance peak frequency and bandwidth. is any complex root, then is also the root. Let the resonance peak frequency corresponding to zi be Fi, and the 3dB bandwidth be Bi, then:

[0046]

[0047] Furthermore, in the step S22, constructing the adaptive segmentation algorithm of the framework AUGAM includes the following steps:

[0048] S221. An adaptive segmentation method is proposed. By introducing learnable nodes, the activation function can adaptively adjust its compression method according to the speech characteristics of different frequencies.

[0049] S222, adaptive segmentation also splits long sentences by RMS value, dividing the entire speech signal into segments of similar length.

[0050] Furthermore, in step S221, the adaptive segmentation method includes the following steps:

[0051] S2211, Helium voice high frequency band and low frequency band processing mode is different, so different frequency bands need their own activation

[0052] Active function:

[0053]

[0054] Among them, is the compression ratio learned in training, which is used to control the shape of the compression function.

[0055] Furthermore, in step S222, the adaptive segmentation method includes the following steps:

[0056] S2221. Divide the entire speech signal into segments of similar length. Detect the silent part in the speech by calculating the RMS value of each frame:

[0057]

[0058] Where N is the number of samples per frame and xi is the amplitude value of the i-th sample.

[0059] S2222, by calculating the RMS value of each frame. Once it is detected that the effective sound portion reaches the minimum length since the last slice, and the silent portion is longer than the minimum interval, the audio is separated from the frame with the lowest RMS value in the silent area:

[0060]

[0061] Among them, t cut is the separation point time node, t start is the time when the silent part starts, t end This is when the silent section ends.

[0062] Furthermore, in the step S231, the main loss function in the fusion loss function includes the following steps:

[0063] S2311. The generator loss of the main loss function plays a core role in network training and is mainly responsible for network training updates. Let Q′ be a function representing the target evaluation metric normalized between 0 and 1, where I represents the input of the target evaluation metric. If a pair of helium speech G(x) and its corresponding normal speech y are evaluated, it is necessary to ensure that the behavior of the discriminator network D is similar to Q′, then the objective function of D is:

[0064]

[0065] in, The discriminator loss is used to train the discriminator network to distinguish between generated speech and real speech.

[0066] S2311. The discriminator loss of the main loss function plays a core role in network training, which completely relies on adversarial loss:

[0067]

[0068] in, is the generator loss, which focuses on the quality of the generated speech and the consistency of semantics. s represents the expected score. In this model, setting s to 1 indicates that the original speech is expected.

[0069] Furthermore, in the step S232, the auxiliary loss function in the fusion loss function includes the following steps:

[0070] S2321, auxiliary loss function is used to compare the similarity between two sequences, and to measure the similarity on the time axis by finding the best matching path between the two sequences. There are two sequences X = (x1, x2, ..., x N ) and Y=(y1,y2,...,y M ) where xi and yj represent the elements in the sequence. The goal of the algorithm is to find a path that minimizes the total distance. The auxiliary distance can be expressed as follows:

[0071]

[0072] Among them, (i1,j1),(i2,j2),…,(i k ,j k ) is a sequence of points on the path that satisfy the start and end conditions, and d(x ik ,y jk ) indicates the position (i) in the sequence X and Y k ,j k ) is a distance or similarity measure between elements at .

[0073] Furthermore, the S3 step includes the following steps:

[0074] S31. Build a paragraph evaluation metric based on AMGAN to evaluate paragraph-based similarity.

[0075] S32. Build an isolated word metric based on AMGAN to evaluate the accuracy based on isolated words.

[0076] Furthermore, in the step S31, the paragraph-based evaluation index includes the following steps:

[0077] S311. Using the overall similarity cannot evaluate the details. Therefore, the evaluation of the keyword similarity in the sentence uses the comprehensive similarity proposed in this paper. For the CS score, there is a document D, given a query Q and the keywords contained are, then the BM25 matching score can be defined as:

[0078]

[0079] Among them, f(q i ,D) represents q i The frequency of words in document D, |D| represents the number of words in document D, avgdl represents the average length of all documents in the corpus, ki and b are free parameters.

[0080] IDF(q i ) represents word q i The inverse document frequency is calculated as follows:

[0081]

[0082] Where N is the total number of documents in the corpus, n(q i ) represents the number of documents containing qi.

[0083] Perform the following steps to make the CS score have the function of measuring semantic similarity

[0084] a) Indexing: Keywords in both text A and B are considered as queries, and each other is considered as a document. That is, text A is indexed first.

[0085] b) Query: Take text B as the query and use BM25 to score text A.

[0086] c) Reverse query: index text B, use text A as the query, and use BM25 to score text B.

[0087] d) Comprehensive score: Add these two scores and average them to get a CS score.

[0088] S312, word segmentation is the process of breaking down continuous text into separate words or phrases. Word segmentation can extract the core vocabulary and semantic information in the sentence. Jaccard similarity is used as the evaluation index for isolated word correction of helium speech:

[0089]

[0090] Among them, A and B are two sets, |A∩B| represents the size of the intersection of the two sets, and |A∪B| represents the size of the union of the two sets.

[0091] Compared with the prior art, the present invention has the following beneficial effects:

[0092] 1. The present invention designs and implements a generative adversarial network model, namely AMGAN, which significantly improves the clarity and intelligibility of helium speech through an adaptive speech segmentation algorithm and a fusion loss function.

[0093] 2. At present, preliminary research results have been achieved in the correction of helium speech for paragraphs and isolated words. The AMGAN method performs well in helium speech correction, and its improvement in clarity and intelligibility is significantly better than traditional methods. In addition, the comprehensive similarity index provides a new perspective for the evaluation of speech correction effects, which can more comprehensively reflect the quality of the corrected speech.

[0094] 3. The research results of this invention not only enrich the theoretical basis of the field of helium speech correction, but also provide effective technical support for practical applications. At the same time, it also has certain guiding value for improving the efficiency and safety of deep-sea diving communications, and provides a reference for the future development of helium speech processing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0096] Figure 1 AMGAN network structure schematic diagram of the present invention.

[0097] Figure 2 This is the FBank feature extraction flow chart of the present invention.

[0098] Figure 3 This is a schematic diagram of the impact of different α on LS nodes in the present invention.

[0099] Figure 4 This is a flow chart of CS score calculation in the present invention.

[0100] Figure 5 Schematic diagram of FBank features of sentences in the present invention.

[0101] Figure 6 Schematic diagram of the resonance peak characteristics of sentences in different environments in the present invention.

[0102] Figure 7 This is the CS score heat map of the present invention.

[0103] Figure 8 Schematic diagram of the speech features of isolated words in different environments in the present invention. DETAILED DESCRIPTION

[0104] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0105] Example 1

[0106] See also Figure 1 and Figure 8 The technical solution provided by this embodiment is a helium speech correction method based on a generative adversarial network, comprising the following steps:

[0107] This invention is based on the MetricGAN+ model. On this basis, a generative adversarial network based on adaptive speech is proposed. By introducing a hybrid architecture of adaptive speech segmentation algorithm and fusion loss function, the perceptual quality and semantic consistency of speech are comprehensively considered. The structure of AMGAN is as follows: Figure 1 As shown. In order to process inputs of different lengths, the discriminator of AMGAN uses a CNN network with multiple two-dimensional convolutional layers and a global average pooling layer. Even when speech of different lengths is input at the input, the FBank features and pitch period features of the speech can be fixed at a suitable length after being processed by the discriminator proposed in this article. The generator is a fully convolutional feedforward network with a residual block. The fully connected layers are LeakyReLU nodes and learnable nodes. And before TF mask processing, the downsampling is applied to the mask estimation calculation.

[0108] Specifically, the first step is to extract the Helium Voice FBank process, mainly as follows Figure 2 As shown, and the resonance peak features are extracted.

[0109] a) Pre-emphasis: The speech signal is passed through a high-pass filter to enhance the high-frequency part of the speech signal and keep the signal in the entire frequency band from low frequency to high frequency. After pre-emphasis processing, the spectrum can be obtained with the same signal-to-noise ratio.

[0110] b) Framing: Divide the audio signal into frames of fixed length, and use the frame length as the minimum unit.

[0111] c) Windowing: Adjacent frames need to have overlapping parts, that is, the distance the window moves each time is less than the frame length.

[0112] d) DFT processing: extract a fixed number of features from each frame, and perform discrete Fourier transform (DFT) on the frame signal to obtain the power spectrum estimate of the periodogram. The following is a public statement:

[0113]

[0114] Among them, S(n) represents the time domain signal, S i (n) represents the data of the i-th frame, n is the value range, S i (k) represents the kth complex coefficient of the i-th frame, P i (k) represents the power spectrum of the i-th frame, h(n) is the window function of N points, and k is the length of DFT.

[0115] e) Mel filtering: After DFT, the amplitude of each frequency component is obtained, and the energy of each frequency component is obtained by squaring the amplitude. Since the human ear is sensitive to different frequencies, it is more sensitive to low frequencies and less sensitive to high frequencies. In order to simulate this characteristic of the human ear, the frequency is mapped to the Mel frequency. The relationship between the Mel frequency and the frequency is:

[0116]

[0117] Where k is the scaling factor that converts linear frequency into Mel frequency, and b is the reference point that reflects the frequency resolution characteristics of the auditory system.

[0118] Applying the Mel filter bank on the energy spectrum obtains the FBank feature, the formula is:

[0119]

[0120] In order to extract the resonance peak, it is necessary to establish an electrical transmission simulation model to approximate the human vocal tract as a first-order lossy sound tube. Since changes in ambient pressure and respiratory gas composition do not affect the overall shape of the source spectrum, and the speech spectrum is obtained by multiplying the source spectrum and the transmission spectrum, if the shape of the source spectrum remains unchanged, the amplitude of the helium speech spectrum in the high-frequency range will be significantly reduced. Therefore, pre-emphasis processing must be performed before extraction:

[0121] s′(n)=s(n)-μs(n-1) (5)

[0122] Among them, μ is the pre-emphasis weight, and s represents the original signal.

[0123] Through pre-emphasis processing, the amplitude of the high-frequency component can be increased, and the high-frequency attenuation caused by the expansion of the transmission spectrum can be compensated, so that the amplitude of each resonance peak is more uniform when the resonance peak parameters are extracted.

[0124] Next is the prediction error filtering process, the filter is as follows:

[0125]

[0126] Where p is the order of the prediction error filter, and γi is the filter coefficient of each order. The complex roots of the polynomial represent the resonance peak frequency and bandwidth. is any complex root, then is also the root. Let the resonance peak frequency corresponding to zi be Fi, and the 3dB bandwidth be Bi, then:

[0127]

[0128] Right now:

[0129]

[0130] In helium speech correction, the traditional fixed parameter method is difficult to meet the differentiated processing requirements of speech at different frequencies. At the same time, due to the special acoustic characteristics of helium speech, the processing modes of high-frequency and low-frequency bands of helium speech are different, so different frequency bands need their own activation functions. The activation function with fixed parameters is difficult to cope with such complex situations, thus affecting the effect of speech correction. To solve this problem, this paper proposes an adaptive segmentation method. By introducing learnable nodes, the activation function can adaptively adjust its compression method according to the speech characteristics of different frequencies, namely:

[0131]

[0132] Among them, α is the compression ratio learned during training and is used to control the shape of the compression function.

[0133] The key to the adaptive segmentation algorithm is to introduce learnable parameters so that the activation function can be dynamically adjusted during the training process to better adapt to the speech characteristics of different frequencies. Figure 3 As shown, higher α behaves like a hard threshold, not smooth but more saturated approximation to a binary mask with most output values ​​being either 0 or 1. Lower α behaves more like a linear function.

[0134] In this way, high-frequency and low-frequency speech can be optimized separately. At the same time, after introducing learnable parameters, the activation function can learn the best processing method during the training process, so that it is expected to achieve the best correction effect in different frequency ranges.

[0135] In addition, adaptive segmentation also splits long sentences by the root mean square (RMS) value, dividing the entire speech signal into segments of similar length. By calculating the RMS value of each frame, the silent part of the speech is detected. That is:

[0136]

[0137] Where N is the number of samples per frame and xi is the amplitude value of the i-th sample.

[0138] By calculating the RMS value of each frame. Once it is detected that the effective sound part has reached the minimum length since the last slice, and the silent part is longer than the minimum interval, the audio will be separated from the frame with the lowest RMS value in the silent area. That is:

[0139]

[0140] Among them, t cut is the separation point time node, t start is the time when the silent part starts, t end This is when the silent section ends.

[0141] At the same time, the longer silent parts are processed, deleted or shortened to avoid unnecessary pauses that affect speech continuity. The above operations can effectively reduce the time error caused by the same person speaking at different times, and correspondingly reduce the accuracy impact caused by this time error. Through adaptive segmentation, the speech signal is segmented into short sentences that are easier to process, and can also maintain a high degree of semantic consistency in the subsequent speech conversion and processing process.

[0142] At the same time, in order to better train the model, a fusion loss function is used. Total generator loss function The purpose of this study is to better quantify the difference between two speech segments, so that it can effectively convert helium speech into normal speech while maintaining semantic consistency and perceptual quality. The loss function consists of multiple components, each of which has its own specific purpose and function. The total loss function can be expressed as a weighted combination of these loss functions.

[0143]

[0144] Among them, λ is the weight coefficient of the loss function, are the main loss function and the auxiliary loss function.

[0145] The main loss function plays a core role in network training and is mainly responsible for network training updates. Let Q′ be a function representing the target evaluation metric normalized between 0 and 1, where I represents the input of the target evaluation metric. If a pair of helium speech G(x) and its corresponding normal speech y are evaluated, it is necessary to ensure that the behavior of the discriminator network D is similar to Q′, then the objective function of D is:

[0146]

[0147] in, The discriminator loss is used to train the discriminator network to distinguish between generated speech and real speech.

[0148] The training of the generator network G relies entirely on adversarial loss:

[0149]

[0150] in, It is the generator loss, which focuses on the quality of the generated speech and the consistency of semantics. s represents the expected score. In this model, setting s to 1 means that the original speech is expected.

[0151] The auxiliary loss function is mainly used to solve the time distortion problem caused by factors such as speaking speed, pitch or tone during the training process. In helium speech correction, traditional speech processing methods often perform poorly when dealing with changes in speaking speed and pitch; this is largely because different speaking speeds, pitches and tones will cause time distortions, making it difficult for the model to capture accurate speech features; in order to solve this problem, this paper adds an auxiliary loss function to enhance the model's ability to handle time distortion problems, thereby reducing the errors caused by time distortion. By aligning and adjusting the time dimension, the model can still maintain a high correction accuracy when facing changes in speaking speed, pitch and tone.

[0152] The auxiliary loss function is used to compare the similarity between two sequences and measure the degree of similarity on the time axis by finding the best matching path between the two sequences.

[0153] Suppose there are two sequences X=(x1,x2,...,x N ) and Y=(y1,y2,...,y M ) where xi and yj represent the elements in the sequence. The goal of the algorithm is to find a path that minimizes the total distance. The auxiliary distance can be expressed as follows:

[0154]

[0155] Among them, (i1,j1),(i2,j2),…,(i k ,j k) is a sequence of points on the path that satisfy the start and end conditions, and d(x ik ,y jk ) indicates the position (i) in the sequence X and Y k ,j k ) is a distance or similarity measure between elements at .

[0156] By introducing sequence alignment to minimize the auxiliary distance, the alignment of the converted speech on the time axis is ensured, and the time distortion problem between helium speech and normal speech is also partially solved.

[0157] After the correction, the correction effect needs to be evaluated. Using speech recognition technology to convert speech into text can simplify the comparison process of speech data. Using natural language processing technology to compare the semantic similarity between the generated text and the reference text, the quality of speech conversion can be indirectly evaluated, so that the analysis of the correction algorithm can be focused. Based on this idea, both normal speech and trained speech go through the same speech recognition model to ensure consistency in the speech-to-text conversion process. This unified standard reduces conversion errors and provides a fairer evaluation standard.

[0158] First, the error rate of paragraph evaluation cannot effectively compare the similarity between the keywords of two sentences, and the overall similarity cannot evaluate the details. Therefore, the evaluation of keyword similarity in sentences uses the comprehensive similarity (CS) score proposed in this paper to evaluate.

[0159] For CS score, there is a document D, given a query Q and containing keywords q1,…,q n , then the BM25 matching score can be defined as:

[0160]

[0161] Among them, f(q i ,D) represents the word frequency of qi in document D, |D| represents the number of words in document D, and avgdl represents the average length of all documents in the corpus. ki and b are free parameters.

[0162] IDF(q i ) represents word q i The inverse document frequency is calculated as follows:

[0163]

[0164] Where N is the total number of documents in the corpus, n(q i ) represents the number of documents containing qi.

[0165] At the same time, BM25 is an optimization of the TF-IDF algorithm. In the calculation of word frequency, BM25 limits the impact of the word frequency of the keyword qi in document D on the score. In order to prevent the word frequency from being too large, BM25 sets the upper limit of this value to k1+1.

[0166] Although BM25 itself is not a similarity metric, it can be used to calculate similarity to some extent. For example, if there are two texts A and B and you want to measure their similarity, you can do something like Figure 4 The above steps can make the CS score have the function of measuring semantic similarity.

[0167] a) Indexing: Keywords in both text A and B are considered as queries, and each other is considered as a document. That is, text A is indexed first.

[0168] b) Query: Take text B as the query and use BM25 to score text A.

[0169] c) Reverse query: index text B, use text A as the query, and use BM25 to score text B.

[0170] d) Comprehensive score: Add these two scores and average them to get a CS score.

[0171] The second is the evaluation of isolated words. During diving, communication relies more on complete sentences, but the identification and correction of isolated words are equally important. And the pronunciation of words may be slightly different when reading aloud individual words that contain the same part and reading aloud in complete sentences. This difference comes from the fact that in a sentence, words are affected by the context and the context, and their pronunciation and intonation will be adjusted. When reading words alone, the pronunciation is relatively independent and standard. Therefore, when evaluating isolated words, this subtle difference must be taken into account. Therefore, the isolated words evaluated in this article are mainly divided into two categories, one is the isolated words read directly, and the other is the isolated words obtained from the sentence through word segmentation.

[0172] Word segmentation is the process of breaking down continuous text into separate words or phrases. This is particularly important for Chinese text processing because Chinese text has no obvious word boundaries. Word segmentation can extract the core vocabulary and semantic information in the sentence. At the same time, Chinese-based word segmentation technology is already very mature, so after the sentence is segmented, the isolated words can be corrected and evaluated.

[0173] In view of the above evaluation requirements, Jaccard similarity is used as the evaluation indicator for isolated word correction in helium speech. Jaccard similarity is an evaluation indicator widely used in the field of speech recognition. Compared with the traditional word error rate, Jaccard similarity pays more attention to the overall similarity of words rather than the offset of words, and does not include time relationship. It can well evaluate the correction performance of isolated words.

[0174] Use Jaccard similarity to compare the similarity of isolated words to get a similarity score. Jaccard similarity is expressed as follows:

[0175]

[0176] Among them, A and B are two sets, |A∩B| represents the size of the intersection of the two sets, and |A∪B| represents the size of the union of the two sets.

[0177] However, there is no public dataset for Helium speech processing, so we first need to build a Helium speech dataset.

[0178] In the helium speech dataset constructed in this paper, the collected data comes from the speech read by divers using helium-oxygen gas supply when saturating to a depth of 83m-110m. The dataset is stored in WAV format with a sampling rate of 16kHz.

[0179] The total length of the data set is 326 minutes, read by 9 divers with rich diving experience, using highly sensitive microphones and professional recorders to ensure audio quality. These 9 divers have rich diving experience and can accurately read professional terms and technical instructions. The average reading time of each diver is about 36 minutes. These data are divided into smaller voice segments in subsequent speech processing and analysis to facilitate model training and testing.

[0180] The reading content mainly includes five parts:

[0181] a) Common words: These include words frequently used in daily diving operations. The pronunciation and recognition of these common words are particularly important in underwater environments. They can be used to analyze the pronunciation changes of divers under different pressure and environmental conditions and help optimize the performance of the speech correction system.

[0182] b) Article paragraphs: These include some common review article paragraphs, which cover topics ranging from literary reviews to current affairs reviews. Through these texts, we can study the changes in the voice characteristics of divers under different topics.

[0183] c) Diving terminology: The terminology section includes terms related to diving operations, equipment use, emergency response, etc. Because they involve key instructions and safe operations, the pronunciation characteristics of these professional terms in a high-pressure environment are of great research value.

[0184] d) Daily conversation: The daily conversation texts selected some conversations between simulated divers during work breaks, including greetings, discussing work progress, arranging tasks, etc. This part of the content is helpful for analyzing the speech characteristics in natural conversation situations.

[0185] e) Technical instructions: This section includes detailed descriptions of diving equipment and operating procedures. Divers need to use more formal and technical language when reading these instructions, which places higher requirements on the accuracy of voice conversion.

[0186] Table 1 shows the detailed information of constructing the Helium voice dataset, including the voice type, recording environment, number of people reading the voice, average reading time, diving depth relative to the sea level, and the average number of words in the corresponding audio.

[0187] Table 1 Helium speech dataset information

[0188]

[0189]

[0190] After that, it is trained and evaluated in the dataset. First, the semantic textual similarity score (STS) in the HanLP library is used as the evaluation indicator of the overall similarity of the corrected sentences. Then, the comprehensive similarity score proposed in this paper is used as the evaluation indicator of the sentence keyword similarity. The correction algorithm is comprehensively evaluated from multiple dimensions.

[0191] The experiment was conducted on 12 Chinese text paragraphs in the Helium Voice dataset, by segmenting the paragraphs into several sentences and extracting their features. The feature extraction mainly includes FBank features and formant features. Figure 5 The corresponding FBank features are obtained by extracting FBank features for the helium speech "We know that water is an important component of living things".

[0192] Figure 6 The formant feature comparison of the sentences is shown. Formant is a key feature that reflects the speech spectrum. The comparison results of the formant features of the converted speech and the normal speech also show a high similarity. This further verifies the effectiveness and stability of the AMGAN model in the speech conversion process.

[0193] Figure 7It is a heat map of CS scores. The heat map intuitively shows the distribution of CS scores of different sentences in the same paragraph under normal pressure environment and after model correction. It can be seen from it that the CS scores along the diagonal are significantly higher than others, indicating that the paragraph after AMGAN conversion has a high similarity with the speech in the corresponding normal pressure environment, that is, the keywords of the paragraph corresponding to the normal pressure environment have a high correlation with the sentences in the converted helium-oxygen environment.

[0194] By analyzing 12 groups of text paragraphs in the helium speech dataset, the experimental results are shown in Table 2, which details the STS and CS scores of different natural paragraphs. By comparing these data, it can be found that the conversion effect of the AMGAN model is relatively ideal in most natural paragraphs, showing relatively high STS and CS scores. At the same time, based on multiple comparisons of the CS scores of two identical article paragraphs, this study concludes that when the CS score is above 15, it can indicate that the paragraphs have a high similarity.

[0195] Through the analysis of the above experimental data, it is found that the AMGAN model performs excellently in long speech correction. The STS and CS scores of most natural paragraphs are relatively high, indicating that the conversion effect of the model at the sentence level is good. At the same time, the STS score shows the similarity of the speech before and after conversion in the text correction result, proving that the model can effectively retain the language information of the original speech. And the relatively high CS score indicates that the converted keywords are close to the original normal pressure speech, verifying the practical application value of the model.

[0196] Table 2 STS similarity and CS scores of natural paragraphs

[0197]

[0198]

[0199] Next is the evaluation of isolated words. In the study of isolated words, feature extraction is performed on the input helium speech data. Figure 8 They are the speech features of isolated words in different environments. Among them, abc are the waveform diagram, spectrogram and FBank features of "please me have" under normal pressure, def are the features in the corresponding helium-oxygen environment, and ghi are the features of the speech corrected to normal speech through the AMGAN network. By comparing the features of the speech in the helium-oxygen environment and the normal pressure environment, it can be found that some features of the trained audio are already similar to those of the audio under the normal pressure environment.

[0200] In addition, Table 3 also uses two indicators, STS and CS, for the analysis of sentences. The experimental results show that the AMGAN model also has a good correction effect at the sentence level. Comparing the STS scores and CS scores of different sentences shows that the similarity between sentences is also at a relatively high level, further verifying the conversion ability of the model.

[0201] Table 3 STS similarity and CS scores of natural sentences

[0202]

[0203] Table 4 shows the correction results of 9 groups of isolated words in helium speech evaluated by Jaccard similarity after word segmentation. It can be seen that AMGAN can well restore isolated words in the helium-oxygen environment to normal speech that can be recognized.

[0204] Table 4 Jaccard similarity of isolated words

[0205]

[0206] The experimental results show that overall, the accuracy of all test sets is above 90%, and the worst set of test sets has an accuracy of 93.65%. Through the analysis of the original speech, it is finally believed that the possible reason is that the accent of one of the saturation divers caused inaccurate correction of single words, resulting in the lowest similarity. The AMGAN model showed a high Jaccard similarity on most word lists, indicating that the model speech conversion effect is good. By comparing the word correction results, the performance of the model in correcting isolated words was evaluated, and the accuracy and stability of the model were verified.

[0207] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A helium speech correction method based on generative adversarial network, characterized in that: The following steps are involved: S1. Preprocess the helium voice data, including denoising and pre-emphasis, to reduce the reduction of the voice spectrum amplitude in the high-frequency range and ensure the clarity of the voice signal; S2. Build a dedicated helium speech conversion model, which combines deep learning methods to convert the processed helium speech signal into standard speech; The Helium speech conversion model consists of two parts: the model’s generator and discriminator; The speech feature extraction module extracts the features of the pre-processed speech, including the unique spectral features of the speech; the speech conversion module uses the GAN-based adaptive speech generative adversarial network AMGAN to map the extracted features into normal speech signals, thereby improving the naturalness and clarity of the speech; S3. The preprocessed Helium speech data is input into the speech recognition and semantic similarity evaluation system for processing, and finally two types of results are generated: one is the text output from the speech recognition module, and the other is the score based on semantic similarity. These two types of results together constitute the basis for evaluating the quality of Helium speech correction.

2. The helium speech correction method based on generative adversarial network according to claim 1 is characterized in that: The S1 step includes the following steps: S11, pre-emphasize the input audio sequence, pass the speech signal through a high-pass filter, thereby enhancing the high-frequency part of the speech signal, and keep the signal in the entire frequency band from low frequency to high frequency, and use the same signal-to-noise ratio to obtain the spectrum after pre-emphasis processing; S12, for the pre-emphasized node, divide the audio signal into frames of fixed length, and use the frame length as the minimum unit; S13, the framed data is windowed, and adjacent frames need to have overlapping parts, that is, the distance the window moves each time is less than the frame length.

3. The helium speech correction method based on generative adversarial network according to claim 1 is characterized in that: The S2 step includes the following steps: S21. The framework is based on the adaptive speech-based generative adversarial network AMGAN. AMGAN learns a variety of input features, including FBank features and formant features. AMGAN learns different features and models the audio. S22. Build the adaptive segmentation algorithm of AMGAN. The two parts of the adaptive segmentation algorithm meet the differentiated processing requirements of speech with different frequencies. In the step S22, the adaptive segmentation algorithm of the AUGAM framework is constructed, including the following steps: S221. An adaptive segmentation method is proposed, which introduces learnable nodes to enable the activation function to adaptively adjust its compression method according to the speech characteristics of different frequencies; In the step S221, the adaptive segmentation method includes the following steps: S2211, Helium voice high frequency band and low frequency band processing mode is different, so different frequency bands need their own activation function: Among them, is the compression ratio learned in training, which is used to control the shape of the compression function; S222, adaptive segmentation: splitting the long sentence by the root mean square value, and dividing the entire speech signal into segments of similar length; In the step S222, the adaptive segmentation method includes the following steps: S2221, dividing the entire speech signal into segments of similar length, and detecting the silent part in the speech by calculating the RMS value of each frame: Where N is the number of samples per frame, and xi is the amplitude value of the i-th sample; S2222. By calculating the RMS value of each frame, once it is detected that the effective sound portion reaches the minimum length since the last slice, and the silent portion is longer than the minimum interval, the audio is separated from the frame with the lowest RMS value in the silent area: Among them, t cut is the separation point time node, t start is the time when the silent part starts, t end It is the time when the silent part ends; S23. Construct the fusion loss function of the framework AMGAN. The fusion loss function is to quantify the difference between the two speech segments so that it can convert the helium speech into normal speech. In the step S23, the main loss function in the fusion loss function includes the following steps: S231. The generator loss of the main loss function plays a core role in network training and is mainly responsible for the training and updating of the network. Let Q′ be a function representing the target evaluation metric normalized between 0 and 1, where I represents the input of the target evaluation metric. If a pair of helium speech G(x) and its corresponding normal speech y are evaluated, it is necessary to ensure that the behavior of the discriminator network D is similar to Q′, then the objective function of D is: in, The discriminator loss is used to train the discriminator network to distinguish between generated speech and real speech; S232. The discriminator loss of the main loss function plays a core role in network training, which is completely dependent on adversarial loss: in, is the generator loss, which focuses on the quality of the generated speech and the consistency of semantics. s represents the expected score. In this model, setting s to 1 indicates that the original speech is expected. In the step S231, the auxiliary loss function in the fusion loss function includes the following steps: S2311, auxiliary loss function is used to compare the similarity between two sequences. It measures the similarity on the time axis by finding the best matching path between the two sequences. There are two sequences X = (x1, x2, ..., x N ) and Y=(y1,y2,...,y M ) where xi and yj represent the elements in the sequence, and the goal of the algorithm is to find a path that minimizes the total distance. The auxiliary distance is expressed by the following formula: Among them, (i1,j1),(i2,j2),…,(i k ,j k ) is a sequence of points on the path that satisfy the start and end conditions, and d(x ik ,y jk ) indicates the position (i) in the sequence X and Y k ,j k ) is a distance or similarity measure between elements at .

4. The helium speech correction method based on generative adversarial network according to claim 3 is characterized in that: In the step S21, constructing the multiple input features of AMGAN includes the following steps: S211, extract FBank features from the preprocessed audio to input into the model; S212. After preprocessing the audio, an electrical transmission simulation model is established, the human vocal tract is approximated as a first-order lossy sound tube, and the resonance peak characteristics are extracted for input into the model.

5. The helium speech correction method based on generative adversarial network according to claim 4 is characterized in that: In the step S211, the input features include the following steps: S2111, performing DFT processing on the input data and then performing logarithm processing; S2112, performing Mel filtering on the data after DFT processing.

6. The helium speech correction method based on generative adversarial network according to claim 4 is characterized in that: In the step S212, the input features include the following steps: S2121, establishing an electrical transmission simulation model with the input data; S2122. Extract resonance peaks from the electrical transmission simulation model.

7. The helium speech correction method based on generative adversarial network according to claim 5 is characterized in that: In the step S2111, the data processing includes the following steps: S21111, the windowed data is processed by DFT to extract a fixed number of features from each frame, and the frame signal is processed by discrete Fourier transform to obtain the power spectrum estimation of the periodogram. The formula is as follows Where S(n) represents the time domain signal, S i (n) represents the data of the i-th frame, n is the value range, S i (k) represents the kth complex coefficient of the i-th frame, P i (k) represents the power spectrum of the i-th frame, h(n) is the window function of N points, and k is the length of DFT.

8. The helium speech correction method based on generative adversarial network according to claim 5 is characterized in that: In the step S2112, the data processing includes the following steps: S21121. Mel filtering is performed on the data after DFT processing. The amplitude of each frequency component is obtained after DFT. The energy of each frequency component is obtained by squaring the amplitude. Since the human ear has different sensitivities to different frequencies, it is more sensitive to low frequencies and less sensitive to high frequencies. In order to simulate this characteristic of the human ear, the frequency is mapped to the Mel frequency. The relationship between the Mel frequency and the frequency is as follows: Where k is the scaling factor, which converts the linear frequency into the Mel frequency, and b is the reference point reflecting the frequency resolution characteristics of the auditory system; Applying the Mel filter bank on the energy spectrum obtains the FBank feature, the formula is: Where Y is the corresponding feature.

9. The helium speech correction method based on generative adversarial network according to claim 5 is characterized in that: In the step S2121, the data processing includes the following steps: S21211. If the shape of the source spectrum remains unchanged, the amplitude of the helium speech spectrum in the high-frequency range will be significantly reduced, and pre-emphasis processing should be performed before extraction: s′(n)=s(n)-μs(n-1) (5) Among them, μ is the pre-emphasis weight, s represents the original signal; Next is the prediction error filtering process, the filter is as follows: Where p is the order of the prediction error filter, γi is the filter coefficient of each order, and the complex root of the polynomial represents the resonance peak frequency and bandwidth. is any complex root, then is also the root. Let the resonance peak frequency corresponding to zi be Fi and the 3dB bandwidth be Bi, then: 。 10. The helium speech correction method based on generative adversarial network according to claim 1 is characterized in that: The S3 step includes the following steps: S31, build a paragraph evaluation index based on AMGAN framework to evaluate paragraph-based similarity; In the step S31, the paragraph-based evaluation index includes the following steps: S311. Since the overall similarity cannot evaluate the details, the proposed comprehensive similarity is used to evaluate the similarity of keywords in the sentence. For the CS score, there is a document D, a query Q and the keywords contained are, then the BM25 matching score is defined as: Among them, f(q i ,D) represents q i The frequency of words in document D, |D| represents the number of words in document D, avgdl represents the average length of all documents in the corpus, ki and b are free parameters; IDF(q i ) represents word q i The inverse document frequency is calculated as follows: Where N is the total number of documents in the corpus, n(q i ) represents the number of documents containing qi; Perform the following steps to make the CS score a measure of semantic similarity: a) Indexing: Consider the keywords in text A and B as queries and each other as documents, and index text A first; b) Query: Take text B as the query and use BM25 to score text A; c) Reverse query: index text B, use text A as the query, and use BM25 to score text B; d) Comprehensive score: These two scores are added together and averaged to obtain a CS score; S312, word segmentation is the process of breaking down continuous text into separate words or phrases. Word segmentation can extract the core vocabulary and semantic information in the sentence. Jaccard similarity is used as the evaluation index for isolated word correction of helium speech: Among them, A and B are two sets, |A∩B| represents the size of the intersection of the two sets, and |A∪B| represents the size of the union of the two sets; S32. Build an isolated word metric based on AMGAN to evaluate the accuracy based on isolated words.

Images