Early warning sound design method and device based on IP sound synthesis, equipment and medium

Abstract

The invention discloses an early warning sound design method and device based on IP sound synthesis, equipment and a storage medium. The method comprises the following steps: acquiring effective audio data; carrying out preprocessing on the obtained audio data; converting the preprocessed audio data into phonetic notation symbols and phonemes aligned with the voice, and extracting characteristics such as duration, pitch, loudness and the like as a training data set; inputting the training data set into a model matched with a speech synthesis requirement in advance, and training the speech synthesis model to obtain a trained target speech synthesis model; deploying the speech synthesis model into a deduction part of a preset framework, and inputting characters to generate and synthesize IP speech; and the generated synthesized IP voice is stored in the voice playing device meeting the requirement for alarm broadcasting. Compared with the prior art, by introducing the IP sound synthesis technology, the generated early warning sound has distinct personalized characteristics, the attention of tourists can be attracted, and the attention of early warning information is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of amusement equipment, and in particular to a warning sound design method, device, equipment and storage medium based on IP sound synthesis. Background Art

[0002] With the rapid development of the amusement equipment industry, faults and hidden dangers that occur during the operation of amusement equipment have always been a safety issue that the park attaches great importance to. Various inspections and overall performance evaluations of safety fault troubleshooting of amusement equipment are carried out regularly. However, during the operation of amusement equipment, when users violate the safety warning line and the equipment is about to reach its load limit, the on-site operating personnel are unable to remind customers in time, and the risk factor will increase significantly.

[0003] Traditional warning sounds usually use a single, fixed audio file, which lacks personalization and interest, and is difficult to attract tourists' attention. Most of them are warnings and forecast reminders for operators, and there are fewer warnings for users and on-site operators. Summary of the invention

[0004] The purpose of the present invention is to provide a warning sound design method, device, equipment and storage medium based on IP sound synthesis, aiming to solve the technical problems that the warning sounds of existing amusement parks are relatively single and there are few warnings for users and on-site operators.

[0005] In order to solve the above technical problems, the purpose of the present invention is achieved through the following technical solutions:

[0006] The first aspect of the present application provides a warning sound design method based on IP sound synthesis, the method comprising:

[0007] Get valid audio data;

[0008] Preprocess the acquired audio data;

[0009] Convert the preprocessed audio data into phonetic symbols and phonemes aligned with speech, and extract features such as duration, pitch, and loudness as training data sets;

[0010] Input the training data set into a model that matches the speech synthesis requirements in advance, train the speech synthesis model, and obtain a trained target speech synthesis model;

[0011] Deploy the speech synthesis model to the deduction part of the preset framework, input text to generate synthesized IP voice;

[0012] The generated synthetic IP voice is stored in a voice playback device that meets the requirements for alarm broadcasting.

[0013] In a possible implementation, the preprocessing of the acquired audio data includes:

[0014] Based on the architecture of convolutional neural network or transformer, it automatically learns the complex features of audio information, converts audio signals from time domain to frequency domain, and then realizes dry sound extraction;

[0015] The audio after dry sound extraction is processed and optimized using the BLSM algorithm to improve the quality of the audio signal;

[0016] Process long audio in chunks;

[0017] Perform noise reduction on the long audio after segmentation.

[0018] In a possible implementation, the BLSM algorithm is:

[0019] [d(nL),d(nL+1),d(nL+2),...d(n(L+1)-1] T [L*1],

[0020] A nL =[X n ,X n+1 ,...X n+L ][N*L],

[0021] error:

[0022] The weight update formula is: W n+1 =W n +μA nl e nl / L,

[0023] Among them, [d(nL),d(nL+1),d(nL+2),...d(n(L+1)-1] T is the expected signal, is the input signal, w n is the current filter weight, is the actual output signal, μ is the learning rate, A nl is the transpose of the input signal matrix, e nl / L is the mean of the error vector.

[0024] In a possible implementation, the preprocessed audio data is converted into phonetic symbols and phonemes aligned with speech, and features such as duration, pitch, and loudness are extracted as a training data set, including: extracting phoneme features using the MFCC (Mel-Frequency Cepstral Coefficients) formula, and the MFCC formula is:

[0025] MFCC = DCT(log(|STFT(x)| 2 )),

[0026] Where x is the audio signal; STFT is short-time Fourier transform, log is logarithmic transform; DCT is discrete cosine transform, D(i,j)=d(i,j)+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}, where D(i,j) is the element of the cumulative distance matrix; d(i,j) is the distance between the current frames (usually Euclidean distance is used).

[0027] In a possible implementation, inputting the training data set into a model pre-matched with speech synthesis requirements, training the speech synthesis model, and obtaining a trained target speech synthesis model includes:

[0028] Use the VITS (Variational Inference Text-to-Speech) model as the speech synthesis model;

[0029] Combining the VITS model with a variational autoencoder for variational inference;

[0030] Generate adversarial training (GAN) to make the generated data close to the real data;

[0031] The speech model is trained based on the alignment estimate generated by variational inference and the generative adversarial training method.

[0032] In a possible implementation, estimating the temporal alignment between text and speech based on variational inference includes:

[0033] Mapping the input text features to a distribution in a latent space through the variational autoencoder;

[0034] Using a decoder to sample latent variables from the distribution and generate speech features;

[0035] Use the flow model (Flow) to perform variational inference in the latent variable space and learn the potential distribution of latent variables;

[0036] The latent variables are mapped to Mel spectrogram representation through the decoder;

[0037] According to the input text features, the number of speech frames corresponding to each feature is predicted and time-series aligned with the generated Mel spectrogram.

[0038] In a possible implementation, the method of using Generative Adversarial Networks (GAN) to make the generated data close to the real data includes:

[0039] Adversarial training is performed through GAN consisting of a generator and a discriminator;

[0040] Generate data using a generator;

[0041] Distinguish generated data from real data through the discriminator;

[0042] Through mutual confrontation and training between the generator and the discriminator, the generator is able to generate fake data that is realistic enough to deceive the discriminator.

[0043] A second aspect of the present application provides a warning sound design device based on IP sound synthesis, the device comprising:

[0044] A sampling unit, used to obtain valid audio data;

[0045] A preprocessing unit, used for preprocessing the acquired audio data;

[0046] A conversion unit, used to convert the preprocessed audio data into phonetic symbols and phonemes aligned with speech, and extract features such as duration, pitch, loudness, etc. as a training data set;

[0047] A model training unit, used for inputting the training data set into a model pre-matched with the speech synthesis requirement, training the speech synthesis model, and obtaining a trained target speech synthesis model;

[0048] The speech deduction unit is used to deploy the speech synthesis model into the deduction part of the preset framework and input text to generate synthesized IP speech;

[0049] The alarm broadcast unit is used to store the generated synthetic IP voice into a voice playback device that meets the requirements for alarm broadcasting.

[0050] A third aspect of the present application provides a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0051] A fourth aspect of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method described in the first aspect above can be implemented.

[0052] The beneficial effects of the present invention compared with the prior art are as follows: the present invention provides a warning sound design method based on IP sound synthesis, including: obtaining effective audio data; preprocessing the obtained audio data; converting the preprocessed audio data into phonetic symbols and phonemes aligned with speech, extracting features such as duration, pitch, loudness, etc. as a training data set; inputting the training data set into a model pre-matched with the speech synthesis requirements, training the speech synthesis model, and obtaining a trained target speech synthesis model; deploying the speech synthesis model into the deduction part of the preset framework, inputting text to generate a synthesized IP voice; storing the generated synthesized IP voice in a voice playback device that meets the requirements for alarm broadcasting. Compared with the prior art, the present invention introduces IP sound synthesis technology, and the generated warning sound has distinct personalized characteristics, which can attract the attention of tourists and improve the attention of warning information; and the warning sound design is flexible and diverse, and various fun elements can be integrated to enhance the tourists' play experience; the warning sound implementation method is simple and easy, and can be widely used in various types of amusement equipment to improve the safety of amusement equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0054] Figure 1 Schematic diagram of the process of the warning sound design method based on IP sound synthesis provided by the embodiment of the present invention Figure 1 ;

[0055] Figure 2 Schematic diagram of the process of the warning sound design method based on IP sound synthesis provided by the embodiment of the present invention Figure 2 ;

[0056] Figure 3 Schematic diagram of the process of the warning sound design method based on IP sound synthesis provided by the embodiment of the present invention Figure 3 ;

[0057] Figure 4 A schematic diagram of the VITS model architecture structure of the warning sound design method based on IP sound synthesis provided in an embodiment of the present invention;

[0058] Figure 5 Schematic diagram of the process of the warning sound design method based on IP sound synthesis provided by the embodiment of the present invention Figure 4 ;

[0059] Figure 6Schematic diagram of the process of the warning sound design method based on IP sound synthesis provided by the embodiment of the present invention Figure 5 ;

[0060] Figure 7 A schematic block diagram of a warning sound design device based on IP sound synthesis provided by an embodiment of the present invention;

[0061] Figure 8 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0064] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0065] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0066] The embodiments of the present application provide a method, apparatus, device and storage medium for designing an early warning sound based on IP sound synthesis.

[0067] Figure 1 This is a schematic diagram of the process of the warning sound design method based on IP sound synthesis provided in the embodiment of the present application Figure 1 .like Figure 1 As shown, the method includes the following steps S110-S160.

[0068] S110, obtaining valid audio data;

[0069] Specifically, under normal circumstances, the effective duration of the speech corpus for speech synthesis is about 12 hours or more, each segment of speech is about 10-16 words, and the sampling format is generally required to be uncompressed PCMWAV format, with a sampling rate of 48KHz, 16bit, consistent timbre, volume, and speaking speed, without drift or clipping; the accuracy of the annotation file is not less than 99.8%; the proportion of phoneme boundary errors greater than 10ms is less than 1%; the syllable boundary accuracy is greater than 98%; only in this way can the requirements of the number, type, pitch, sound connection and rhythm of the syllable phonemes of the corpus audio be met. If conditions permit, a professional recording studio environment can be used. The recording environment and equipment must remain unchanged before the recording is completed, and the environmental noise must not be lower than 35dB.

[0070] S120, preprocessing the acquired audio data;

[0071] Specifically, for audio obtained in a non-professional environment, the audio needs to be processed to meet specific conditions.

[0072] S130, converting the preprocessed audio data into phonetic symbols and phonemes aligned with the speech, and extracting features such as duration, pitch, loudness, etc. as a training data set;

[0073] Specifically, the conversion of audio to phonetic symbols and phonemes includes using the corresponding dictionary and G2P (Grapheme-to-Phoneme) model to convert the graphemes in the text corresponding to the audio (which can be transcribed by ASR) into phonemes; using a speech alignment tool (such as Montreal ForceAligner, MFA) to align the preprocessed audio data and the converted phonemes to ensure that each phoneme accurately corresponds to the corresponding part in the audio. Secondly, it is also necessary to extract corresponding features from the phonemes, including duration, pitch, loudness, etc. Among them, the duration feature is to extract the duration information of each phoneme based on the alignment result, and it is realized by calculating the start time and end time of each phoneme in the audio; pitch and loudness features: use the relevant functions in the audio processing library (such as the functions in librosa) to extract the spectral features of the audio signal, and then calculate the pitch (fundamental frequency) and loudness (amplitude or energy). The pitch feature can be obtained by analyzing the peak position in the spectrum graph, and the loudness feature can be obtained by calculating the amplitude or energy of the spectrum graph; and when extracting features, it is very important to consider the contextual relationship between phonemes, which is achieved by analyzing the features of adjacent phonemes, or using more complex models (such as deep learning models) to capture this relationship. The extracted features and the corresponding audio file path, phonetic symbols, aligned phonemes and other information are organized in a specified format (such as "audio path|start time|phonetic symbols and phoneme sequence"). In this embodiment, the content of the data set is in the form of " "The first part indicates the path where the audio is located, and the second part indicates the specific pronunciation and phonemes of the audio.

[0074] S140, inputting the training data set into a model pre-matched with speech synthesis requirements, training the speech synthesis model, and obtaining a trained target speech synthesis model;

[0075] Specifically, the above-mentioned phoneme data is used to divide the training set and the test set, the training configuration file is adjusted, and the speech model training is performed based on the IP Voice Utilization (VITS) conditional variational autoencoder, the alignment estimation generated from variational inference, and the generative adversarial training method.

[0076] S150, deploying the speech synthesis model to the deduction part of the preset framework, inputting text to generate synthesized IP speech;

[0077] Specifically, based on the need to synthesize IP voice, the model trained in step S140 is used to input the text into the voice model, and then decoded into new voice after text encoding, to generate voice segments related to the warning issued by the IP.

[0078] S160: The generated synthesized IP voice is stored in a voice playback device that meets the requirements for alarm broadcasting.

[0079] Specifically, by playing the alarm information before the amusement equipment is operated, the tourists in the amusement equipment can be reminded in advance and the occurrence of danger can be reduced.

[0080] like Figure 2 As shown, in a more specific embodiment, executing step S120 further specifically includes executing steps S1201-S1204.

[0081] S1201, based on the architecture of convolutional neural network or transformer, automatically learn the complex features of audio information, convert the audio signal from time domain to frequency domain, and then realize dry sound extraction;

[0082] Specifically, step S1201 includes selecting a convolutional neural network (CNN) or a transformer architecture as the basis of the model. Among them, CNN is good at capturing local features, while Transformer is good at processing long-distance dependencies. By using a large number of annotated audio data sets for model training, the model can automatically learn the complex features of audio information. Conversion from time domain to frequency domain, pre-processing of input audio signals, such as denoising, standardization, etc.; use algorithms such as fast Fourier transform (FFT) to convert audio signals from time domain to frequency domain in order to better capture the spectral characteristics of the audio. Dry sound extraction: In the frequency domain, the features learned by the model are used to extract dry sound, which usually involves masking of the spectrogram to separate the target sound (dry sound) and background sound; use the inverse fast Fourier transform (IFFT) to convert the extracted dry sound from the frequency domain back to the time domain to obtain the final dry sound signal.

[0083] S1202, processing and optimizing the audio signal after the dry sound extraction using the BLSM algorithm to improve the quality;

[0084] Specifically, BLSM (Bidirectional Long Short-Term Memory) is an improved recurrent neural network (RNN) architecture that can capture bidirectional dependencies in audio signals. The audio signal after dry sound extraction is input into the BLSM algorithm for further processing and optimization. The BLSM algorithm can perform noise reduction, echo removal, and clarity enhancement on audio signals to improve audio quality. According to specific needs, the parameters in the BLSM algorithm can be adjusted to obtain the best processing effect.

[0085] S1203, processing the long audio in blocks;

[0086] Specifically, for long audio signals, a block processing strategy is adopted. This can reduce the memory usage during processing and improve processing efficiency; according to the characteristics of the audio signal and processing requirements, the appropriate block size and overlap rate are selected. The long audio signal is divided according to the selected block size and overlap rate to obtain multiple audio blocks, and each audio block is independently processed and analyzed, such as feature extraction, noise reduction, etc.

[0087] S1204: Perform noise reduction processing on the long audio after being divided into blocks.

[0088] Specifically, each audio signal after being divided is subjected to noise reduction processing to remove the noise components in the audio and improve the clarity and intelligibility of the audio. According to the characteristics of the noise reduction algorithm and the parameter settings, the noise reduction intensity and processing effect are adjusted to achieve the best noise reduction effect. After the noise reduction processing of all audio blocks is completed, they can also be merged in the original order to obtain the processed long audio signal, and the necessary post-processing, such as smoothing and de-jittering, is performed on the merged audio signal to further improve the audio quality.

[0089] In this embodiment, the BLSM algorithm is:

[0090] [d(nL),d(nL+1),d(nL+2),...d(n(L+1)-1] T [L*1],

[0091] A nL =[X n ,X n+1 ,...X n+L ][N*L],

[0092] error:

[0093] The weight update formula is: W n+1 =W n +μA nl e nl / L,

[0094] Among them, [d(nL),d(nL+1),d(nL+2),...d(n(L+1)-1] T is the expected signal, is the input signal, w n is the current filter weight, is the actual output signal, μ is the learning rate, A nl is the transpose of the input signal matrix, e nl / L is the mean of the error vector.

[0095] Specifically, the BLSM algorithm in this embodiment is applied to echo cancellation. The algorithm first collects a set of data, which includes L sampling points, similar to batch processing in neural networks, where L can be regarded as the batch size. For these L sampling points, the algorithm calculates the mean of the error between the expected signal (usually a clean sound signal) and the actual output signal (a signal containing echo or noise), and then updates the weights. After processing each L sample points, the algorithm updates the weights using the above formula; then, the algorithm continues to collect the next set of L sample points and repeats the above process until a predetermined number of iterations is reached or a certain stopping criterion is met (such as the error is less than a certain threshold).

[0096] In this embodiment, executing step S130 further includes: extracting phoneme features using the MFCC (Mel-Frequency Cepst ral Coefficients) formula, where the MFCC formula is:

[0097] MFCC = DCT(log(|STFT(x)| 2 )),

[0098] Where x is the audio signal; STFT is short-time Fourier transform, log is logarithmic transform; DCT is discrete cosine transform, D(i,j)=d(i,j)+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}, where D(i,j) is the element of the cumulative distance matrix; d(i,j) is the distance between the current frames (usually Euclidean distance is used).

[0099] Specifically, the MFCC algorithm includes the following steps: 1) Preprocessing: preprocess the audio signal, such as framing, windowing and pre-emphasis; 2) Short-time Fourier transform (STFT): perform Fourier transform on each frame of the signal to obtain a spectrum; 3) Calculate the power spectrum: square the amplitude of the spectrum to obtain the power spectrum; 4) Apply Mel filter bank: filter the power spectrum through a set of Mel-scale triangular filters to obtain a Mel spectrum; 5) Take the logarithm: take the logarithm of the Mel spectrum to obtain the logarithmic Mel spectrum; 6) Discrete cosine transform (DCT): perform discrete cosine transform on the logarithmic Mel spectrum to obtain the MFCC coefficients.

[0100] like Figure 3 As shown, in a more specific embodiment, executing step S140 further specifically includes executing steps S1401-S1405.

[0101] S1401. Apply the VITS (Variational Inference Text-to-Speech) model as a speech synthesis model.

[0102] S1402, combining the VITS model with the variational autoencoder for variational inference;

[0103] S1403, estimating the temporal alignment between text and speech according to variational inference;

[0104] S1404, making the generated data close to the real data through a generative adversarial network (GAN) method;

[0105] S1405. Perform speech model training based on the alignment estimation generated by variational inference and the generative adversarial training method.

[0106] Specifically, Figure 4 As shown in the figure, the training process of the VITS model is an end-to-end process that combines a generative model based on variational inference and text-to-speech conversion technology. Specifically, the training process of the VITS model can be summarized into the following key steps:

[0107] 1) Input text feature mapping: First, the model accepts input text features. These text features usually include representations of text such as characters, words, or phonemes. The encoder part of the variational autoencoder (VE) maps these input text features to a distribution in a latent space. This latent space can be regarded as a "thinking space" that contains all the necessary information required for text-to-speech conversion. Specifically, in VAE, the speech data x conforms to the complex distribution p θ (X), while directly modeling p θ (X) is more difficult, so VAE introduces a simple prior distribution p θ (Z) (usually a standard normal distribution), starting with a simple distribution p θ (Z) to obtain a latent variable z, and then use a neural network (usually called a decoder or generator) to map z back to the data space to generate a data point x. In VAE, the posterior distribution p(z|x) = Ν(z; μ(x), σ 2 (x)) is approximated as a Gaussian distribution, μ(x) and σ 2 (x) is the mean and variance generated by another neural network (usually called the encoder). In this way, VE can learn an effective mapping from simple distribution to complex data distribution.

[0108] 2) Latent space sampling: In the latent space, the model does not directly determine a unique point to represent the input text, but represents it with a distribution. This can increase the diversity and naturalness of the generated speech. The decoder samples a latent variable from this distribution. This latent variable carries all the information needed to generate speech.

[0109] 3) Speech feature generation: The decoder uses this latent variable to generate speech features. These speech features can be spectral features (such as Mel spectrograms) or other features that can be converted into audio signals. In this process, the role of the decoder is to map the latent variables in the latent space back to the space of audio features.

[0110] 4) Variational Inference: The Flow model performs variational inference in the latent variable space and learns the potential distribution of latent variables. This helps the model understand the structure of the latent space more accurately and improves the sampling process. Through variational inference, the model is able to gradually optimize the representation of the latent space, making it more capable of capturing the complex relationships of text-to-speech conversion.

[0111] 5) Mel spectrogram generation: The decoder finally maps the latent variables to a Mel spectrogram representation. Mel spectrogram is a spectral representation method that is more suitable for representing the characteristics of the human auditory system. This Mel spectrogram can be regarded as an intermediate representation of speech, which can be further converted into an audio signal.

[0112] 6) Model training: During the training process, the model calculates the loss by comparing the generated Mel spectrogram with the Mel spectrogram of the real speech. Through the back-propagation algorithm, the model continuously updates its parameters to minimize this loss function. The training process continues until the model can generate high-quality speech, that is, the generated Mel spectrogram is close enough to the Mel spectrogram of the real speech.

[0113] 7) Get the trained target speech synthesis model: After a large number of training iterations, the model will reach a stable state, at which point it can generate high-quality, natural and fluent speech. This trained model is the target speech synthesis model, which can be used to convert text to speech.

[0114] like Figure 5 As shown, in a more specific embodiment, executing step S1403 further specifically includes executing steps S14031-S14035.

[0115] S14031. Mapping input text features to a distribution in a latent space through a variational autoencoder;

[0116] S14032, using a decoder to sample latent variables from the distribution and generate speech features;

[0117] S14033. Use the flow model (Flow) to perform variational inference in the latent variable space and learn the potential distribution of latent variables;

[0118] S14034, mapping the latent variables into a Mel spectrogram representation through a decoder;

[0119] S14035. According to the input text features, predict the number of speech frames corresponding to each feature, and perform time alignment with the generated Mel spectrogram.

[0120] Specifically, in the VITS model, alignment estimation refers to the temporal correspondence between text and speech. This alignment is crucial for speech synthesis, as it ensures that the generated speech is temporally consistent with the input text content. Variational inference is a key component in the VITS model, which is used to estimate the alignment between text and speech. Specifically, the model maps the input text features to a latent space through the encoder part of a variational autoencoder (VAE) and learns a distribution in this space. The model then uses this distribution to generate latent variables, which are mapped to speech features through a decoder. In this process, variational inference helps the model understand the underlying relationship between text and speech, thereby producing accurate alignment estimates.

[0121] like Figure 6 As shown, in a more specific embodiment, executing step S1404 further specifically includes executing steps S14041-S14044.

[0122] S14041. Adversarial training is performed through a GAN consisting of a generator and a discriminator.

[0123] S14042, generating data using a generator;

[0124] S14043, distinguishing generated data from real data through a discriminator;

[0125] S14044. Through mutual confrontation and training between the generator and the discriminator, the generator is able to generate realistic fake data that is sufficient to deceive the discriminator.

[0126] Specifically, the VITS model training mainly uses GAN (Generative Adversarial Networks) to generate competitive networks, which can make the generated data as similar as possible to the real data through training. D stands for discriminator, G stands for generator, and L stands for D =E[(D(x)-1) 2 +(D(G(z))-0) 2 ], L D =E[(D(x)-1) 2 +λ||xG(z)||, the above formulas are two loss functions of the discriminator, which are used to distinguish real data from fake data.

[0127] Figure 7 1 is a schematic block diagram of a warning sound design device based on IP sound synthesis provided by an embodiment of the present application. Figure 7As shown, corresponding to the above-mentioned warning sound design method based on IP sound synthesis, the present application also provides a warning sound design device 200 based on IP sound synthesis. The warning sound design device 200 based on IP sound synthesis includes a unit for executing the above-mentioned warning sound design method based on IP sound synthesis, and the device can be configured in a desktop computer, a tablet computer, a laptop, and other terminals.

[0128] Specifically, see Figure 7 , the warning sound design device 200 based on IP sound synthesis includes:

[0129] A sampling unit 210, used to obtain valid audio data;

[0130] A preprocessing unit 220, configured to preprocess the acquired audio data;

[0131] A conversion unit 230, for converting the preprocessed audio data into phonetic symbols and phonemes aligned with speech, and extracting features such as duration, pitch, loudness, etc. as a training data set;

[0132] A model training unit 240 is used to input the training data set into a model pre-matched with the speech synthesis requirement, train the speech synthesis model, and obtain a trained target speech synthesis model;

[0133] The speech deduction unit 250 is used to deploy the speech synthesis model into the deduction part of the preset framework and input text to generate synthesized IP speech;

[0134] The alarm broadcast unit 260 is used to store the generated synthesized IP voice into a voice playback device that meets the requirements for alarm broadcasting.

[0135] In addition, more specifically, the preprocessing unit 220 also includes: based on the architecture of a convolutional neural network or a transformer, automatically learning the complex features of audio information, converting the audio signal from the time domain to the frequency domain, and then realizing dry sound extraction; processing and optimizing the audio signal after dry sound extraction using the BLSM algorithm to improve the quality; performing block processing on long audio; and performing noise reduction on the long audio after block processing.

[0136] The conversion unit 230 includes: applying a VITS (Variational Inference Text-to-Speech) model as a speech synthesis model; combining the VITS model with a variational autoencoder for variational inference; estimating the temporal alignment between text and speech based on variational inference; making the generated data close to the real data through a generative adversarial training (GAN) method; and training the speech model based on the alignment estimation generated by variational inference and the generative adversarial training method.

[0137] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned warning sound design device based on IP sound synthesis and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.

[0138] The above-mentioned warning sound design device 200 based on IP sound synthesis can be implemented in the form of a computer program. The computer program can be used in Figure 8 Runs on the computer device shown.

[0139] See also Figure 8 , Figure 8 300 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 300 may be a terminal or a server, wherein the terminal may be an electronic device with communication functions such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The server may be an independent server or a server cluster composed of multiple servers.

[0140] See also Figure 8 The computer device 300 includes a processor 302 , a memory and a network interface 305 connected via a system bus 301 , wherein the memory may include a non-volatile storage medium 303 and an internal memory 304 .

[0141] The non-volatile storage medium 303 can store an operating system 3031 and a computer program 3032. The computer program 3032 includes program instructions, and when the program instructions are executed, the processor 302 can execute a warning sound design method based on IP sound synthesis.

[0142] The processor 302 is used to provide computing and control capabilities to support the operation of the entire computer device 300 .

[0143] The internal memory 304 provides an environment for the operation of the computer program 3032 in the non-volatile storage medium 303. When the computer program 3032 is executed by the processor 302, the processor 302 can execute a warning sound design method based on IP sound synthesis.

[0144] The network interface 305 is used to communicate with other devices over the network. Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 300 to which the solution of the present application is applied. The specific computer device 300 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] It should be understood that in the embodiment of the present application, the processor 302 may be a central processing unit (CPU), and the processor 302 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0146] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0147] Therefore, the present application also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the following steps:

[0148] Get valid audio data;

[0149] Preprocess the acquired audio data;

[0150] Convert the preprocessed audio data into phonetic symbols and phonemes aligned with speech, and extract features such as duration, pitch, and loudness as training data sets;

[0151] Input the training data set into a model that matches the speech synthesis requirements in advance, train the speech synthesis model, and obtain a trained target speech synthesis model;

[0152] Deploy the speech synthesis model to the deduction part of the preset framework, input text to generate synthesized IP voice;

[0153] The generated synthetic IP voice is stored in a voice playback device that meets the requirements for alarm broadcasting.

[0154] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0155] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0156] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0157] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application.

[0159] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A warning sound design method based on IP sound synthesis, characterized in that: The method comprises: Get valid audio data; Preprocess the acquired audio data; Convert the preprocessed audio data into phonetic symbols and phonemes aligned with speech, and extract features such as duration, pitch, and loudness as training data sets; Input the training data set into a model that matches the speech synthesis requirements in advance, train the speech synthesis model, and obtain a trained target speech synthesis model; Deploy the speech synthesis model to the deduction part of the preset framework, input text to generate synthesized IP voice; The generated synthetic IP voice is stored in a voice playback device that meets the requirements for alarm broadcasting.

2. The method for designing an early warning sound based on IP sound synthesis according to claim 1, characterized in that: The preprocessing of the acquired audio data includes: Based on the architecture of convolutional neural network or transformer, it automatically learns the complex features of audio information, converts audio signals from time domain to frequency domain, and then realizes dry sound extraction; The audio after dry sound extraction is processed and optimized using the BLSM algorithm to improve the quality of the audio signal; Process long audio in chunks; Perform noise reduction on the long audio after segmentation.

3. The method for designing an early warning sound based on IP sound synthesis according to claim 2, characterized in that: The BLSM algorithm is: [d(nL),d(nL+1),d(nL+2),...d(n(L+1)-1] T [L*1], A nL =[X n ,X n+1 ,...X n+L ][N*L], error: The weight update formula is: W n+1 =W n +μA nl e nl / L, Among them, [d(nL),d(nL+1),d(nL+2),...d(n(L+1)-1] T is the expected signal, is the input signal, w n is the current filter weight, is the actual output signal, μ is the learning rate, A nl is the transpose of the input signal matrix, e nl / L is the mean of the error vector.

4. The method for designing an early warning sound based on IP sound synthesis according to claim 1, characterized in that: The pre-processed audio data is converted into phonetic symbols and phonemes aligned with speech, and features such as duration, pitch, and loudness are extracted as training data sets, including: extracting phoneme features using the MFCC (Mel-Frequency Cepstral Coefficients) formula, and the MFCC formula is: MFCC=DCT(log(|STFT(x)| 2 )), Where x is the audio signal; STFT is short-time Fourier transform, log is logarithmic transform; DCT is discrete cosine transform, D(i,j)=d(i,j)+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}, where D(i,j) is the element of the cumulative distance matrix; d(i,j) is the distance between the current frames (usually Euclidean distance is used).

5. The method for designing an early warning sound based on IP sound synthesis according to claim 1, characterized in that: The step of inputting the training data set into a model pre-matched with the speech synthesis requirement, training the speech synthesis model, and obtaining a trained target speech synthesis model includes: Use the VITS (Variational Inference Text-to-Speech) model as the speech synthesis model; Combining the VITS model with a variational autoencoder for variational inference; Estimate temporal alignment between text and speech using variational inference; Generate adversarial training (GAN) to make the generated data close to the real data; The speech model is trained based on the alignment estimate generated by variational inference and the generative adversarial training method.

6. The method for designing an early warning sound based on IP sound synthesis according to claim 5, characterized in that: The estimating the temporal alignment between text and speech according to variational inference comprises: Mapping the input text features to a distribution in a latent space through the variational autoencoder; Using a decoder to sample latent variables from the distribution and generate speech features; Use the flow model (Flow) to perform variational inference in the latent variable space and learn the potential distribution of latent variables; The latent variables are mapped to Mel spectrogram representation through the decoder; According to the input text features, the number of speech frames corresponding to each feature is predicted and time-series aligned with the generated Mel spectrogram.

7. The method for designing an early warning sound based on IP sound synthesis according to claim 5, characterized in that: The method of generating adversarial training (GAN) makes the generated data close to the real data, including: Adversarial training is performed through GAN consisting of a generator and a discriminator; Generate data using a generator; Distinguish generated data from real data through the discriminator; Through mutual confrontation and training between the generator and the discriminator, the generator is able to generate fake data that is realistic enough to deceive the discriminator.

8. A warning sound design device based on IP sound synthesis, characterized in that: The device comprises: A sampling unit, used to obtain valid audio data; A preprocessing unit, used for preprocessing the acquired audio data; A conversion unit, used to convert the preprocessed audio data into phonetic symbols and phonemes aligned with speech, and extract features such as duration, pitch, loudness, etc. as a training data set; A model training unit, used for inputting the training data set into a model pre-matched with the speech synthesis requirement, training the speech synthesis model, and obtaining a trained target speech synthesis model; The speech deduction unit is used to deploy the speech synthesis model into the deduction part of the preset framework and input text to generate synthesized IP speech; The alarm broadcast unit is used to store the generated synthetic IP voice into a voice playback device that meets the requirements for alarm broadcasting.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images