Generation method of voice and electroglottography multi-modal data set
By generating multimodal data sets of speech and electroglass diagrams, the problems of insufficient diversity and lack of electroglass diagram information in the prior art are solved, and the performance and robustness of speech recognition and emotion recognition tasks are improved. .
Patent Information
- Application Number
- CN202411275768.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-12
AI Technical Summary
The existing speech data sets are insufficient in diversity and lack of electroglid pattern information synchronized with speech, which limits the generalization ability and robustness of speech recognition systems in extremely low signal-to-noise ratio scenarios and multiple speakers.
By obtaining the original multimodal data read aloud by several preset personnel based on the preset text, performing mixing operations, generating mixed multimodal data under different signal-to-noise ratio conditions, and generating label information for it, based on the preset file storage structure It is stored to form a multimodal data set of voice and electroglass diagrams.
The performance and accuracy of the model in speech recognition and emotion recognition tasks are improved, and the generalization ability, accuracy, robustness and reliability of the model are enhanced.
Smart Images

Figure CN119091884A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of information technology, and in particular relates to a method for generating a multimodal data set of speech and electroglottogram. Background Art
[0002] Speech contains rich information about human speech, including speech content, pitch, speaking speed, identity information, etc., and can be used for tasks such as speech recognition, semantic understanding, voiceprint recognition, and speech emotion recognition. Electroglottogram is a graphical representation of vocal cord vibration. Compared with speech, it has the following advantages: First, electroglottogram data provides a visual representation of vocal cord vibration, which can intuitively show the characteristics and waveform of the tone; second, electroglottogram data is not affected by speech quality and environmental noise, and has high stability and reliability; finally, electroglottogram data has lower storage and transmission costs and is easier to process and analyze. Speech datasets have a wide range of applications in speech recognition, speech synthesis, emotion recognition, and other fields, and can be applied to scenarios such as smart assistants, smart customer service, and speech translation. Electroglottogram datasets can be used not only for speech recognition and emotion recognition, but also for speaker recognition, speech quality assessment, and other fields.
[0003] Existing speech datasets often lack diversity, which limits the generalization and robustness of speech recognition systems in extremely low signal-to-noise ratio scenarios and with multiple speakers. In addition, most datasets lack electroglottogram information synchronized with speech, which limits the possibility of a deep understanding of the speech production mechanism. Summary of the invention
[0004] The purpose of the present invention is to provide a method for generating a multimodal data set of speech and electroglottogram to solve the problems existing in the above-mentioned prior art.
[0005] To achieve the above object, the present invention provides a method for generating a multimodal data set of speech and electroglottogram, comprising:
[0006] Acquiring original multimodal data read aloud by a plurality of preset persons according to preset text, wherein the original multimodal data includes voice data and electroglottogram data;
[0007] Performing a mixing operation on each of the original multimodal data to obtain mixed multimodal data under different signal-to-noise ratio conditions;
[0008] Generate corresponding label information for each of the original multimodal data and each of the mixed multimodal data, store each of the multimodal data and the corresponding label information based on a preset file storage structure, and obtain a multimodal data set.
[0009] Optionally, original multimodal data read aloud by several preset persons according to preset text is obtained, specifically including:
[0010] Numbering and grouping each preset person and each preset text based on a preset division rule to obtain preset person groups and preset text groups with the same number of groups;
[0011] According to the grouping order, each preset personnel group and each preset text group are made to correspond one to one, and each group of preset personnel reads the text according to the preset pause interval to obtain a plurality of dual-channel audio files, wherein the dual-channel audio files include a left-channel audio file and a right-channel electroglottogram file;
[0012] The dual-channel audio files are segmented into text based on a preset pause interval to obtain a plurality of segmented texts, and each segmented text is named to obtain the original multimodal data.
[0013] Optionally, in the process of having each group of preset personnel read the text aloud according to the preset pause interval, each of the dual-channel audio files is collected based on a preset audio collection device, and the preset audio collection device includes a microphone and an electroglottogram instrument.
[0014] Optionally, before performing text segmentation on each of the dual-channel audio files based on a preset pause interval, the method further includes performing noise reduction processing on each of the dual-channel audio files based on an empirical mode decomposition method to obtain a noise-reduced dual-channel audio file, and performing text segmentation based on the noise-reduced dual-channel audio file.
[0015] Optionally, text segmenting is performed on each of the dual-channel audio files based on a preset pause interval, specifically including:
[0016] An audio segment whose sound is lower than a preset decibel threshold and lasts longer than a preset pause interval is regarded as a silent segment, and an audio segment after removing the silent segment from each dual-channel audio file is extracted as the segmented text.
[0017] Optionally, a mixing operation is performed on each of the original multimodal data, specifically including:
[0018] The original multimodal data of two preset persons are randomly selected, the audio length of the selected original multimodal data is uniformly processed, the audio is mixed according to the set signal-to-noise ratio, and named to obtain the mixed multimodal data.
[0019] Optionally, the method further includes storing the mixed multimodal data and the audio information of the corresponding original multimodal data, specifically including:
[0020] Each of the mixed multimodal data is input into a target speech extraction model for speech extraction to obtain original multimodal data corresponding to each of the mixed multimodal data, and the audio information of each of the mixed multimodal data and the corresponding original multimodal data is stored in a CSV file format; wherein the target speech extraction model is constructed based on a recurrent neural network and a gated recurrent network.
[0021] Optionally, the method further includes verifying the availability of the multimodal dataset, specifically including:
[0022] Each multimodal data in the multimodal data set is input into a speech recognition model for classification and recognition, and predicted label information corresponding to each multimodal data is obtained. The predicted label information is compared with the label information of the corresponding multimodal data to obtain a usability verification result, wherein the speech recognition model is constructed based on a neural network model.
[0023] The technical effects of the present invention are:
[0024] By combining speech and electroglottogram data, the present invention can obtain more comprehensive and multi-dimensional information, thereby improving the performance and accuracy of the model in speech recognition and emotion recognition tasks. In training and evaluating speech processing systems, such as speech recognition, speech synthesis, etc., the database can better train and evaluate the model, and can improve the generalization ability, accuracy, robustness and reliability of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0026] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0027] Figure 1 It is a flow chart of the speech electroglottogram dual-modal data acquisition and processing method in an embodiment of the present invention;
[0028] Figure 2 is a flowchart of audio content recording in an embodiment of the present invention; wherein, Figure 2 (a) is a continuous recording process within a group in an embodiment of the present invention. Figure 2 (b) is the inter-group rest and recording process in an embodiment of the present invention;
[0029] Figure 3 is a flow chart of bimodal data set segmentation in an embodiment of the present invention;
[0030] Figure 4 This is a flow chart of a target speech extraction model in an embodiment of the present invention;
[0031] Figure 5 is a flow chart of a speech recognition model in an embodiment of the present invention;
[0032] Figure 6 A graph showing word error rate changes during a test in an embodiment of the present invention;
[0033] Figure 7 A flowchart for generating a multimodal data set in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as limiting the present invention, but should be understood as a more detailed description of certain aspects, features, and embodiments of the present invention.
[0035] It should be understood that the terms described in the present invention are only for describing special embodiments and are not intended to limit the present invention. In addition, for the numerical range in the present invention, it should be understood that each intermediate value between the upper and lower limits of the scope is also specifically disclosed. Each smaller range between the intermediate value in any stated value or stated range and any other stated value or intermediate value in the described range is also included in the present invention. The upper and lower limits of these smaller ranges can be independently included or excluded in the scope.
[0036] Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art generally understood by the present invention. Although the present invention describes only preferred methods, any method similar or equivalent to that described herein may also be used in the implementation or testing of the present invention. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of a conflict with any incorporated document, the content of this specification shall prevail.
[0037] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments of the present invention description without departing from the scope or spirit of the present invention. Other embodiments derived from the present invention description will be apparent to those skilled in the art. The present application description and examples are exemplary only.
[0038] The words “include,” “including,” “have,” “contain,” etc. used in this article are open-ended terms, meaning including but not limited to.
[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0040] Embodiment 1
[0041] like Figure 1 - Figure 7 As shown, in this embodiment, a method for generating a multimodal data set of speech and electroglottogram is provided, comprising: obtaining original multimodal data read aloud by several preset persons according to preset text, wherein the original multimodal data comprises speech data and electroglottogram data; performing a mixing operation on each of the original multimodal data to obtain mixed multimodal data under different signal-to-noise ratio conditions; generating corresponding label information for each of the original multimodal data and each of the mixed multimodal data, storing each multimodal data and the corresponding label information based on a preset file storage structure, and obtaining a multimodal data set.
[0042] This embodiment can obtain more comprehensive and multi-dimensional information by combining speech and electroglottogram data, thereby improving the performance and accuracy of the model in speech recognition and emotion recognition tasks. The emergence of multimodal datasets has made an important contribution to today's technology field and can provide more intelligent and more humane solutions for smart voice assistants, smart medical care, smart homes and other fields.
[0043] In the fields of speech recognition and emotion recognition, the quality and quantity of data sets are crucial to the performance of the model. By providing larger, more organized and more diverse multimodal data sets, we can provide richer and more reliable data support for research and application in the field of artificial intelligence, so as to promote the development and application of artificial intelligence technology.
[0044] This embodiment involves using professional equipment to simultaneously collect speech and electroglottogram signals of one hundred different individuals, and then segmenting and integrating them into a two-channel data set; it can be applied to fields such as speech recognition, speaker recognition, voiceprint recognition, noise suppression and speech enhancement, and can also be used as training sets and test sets for model training, providing important support and assistance for the development and application of related technologies in multiple fields.
[0045] The production, collection and establishment of the speech-electroglottomyogram bimodal Chinese dataset are as follows:
[0046] Step 1: Use hardware equipment to measure the voice and electroglottal feature information of the experimenter;
[0047] Step 1.1: Plan the data set content, select one thousand sentences as the data set content, and sort the one thousand sentences, number them from 000 to 999, and divide them into ten groups in order, with one hundred sentences in each group. It is stipulated that every ten people read a group from beginning to end, and a total of 10,000 sentences will be obtained;
[0048] Step 1.2: Select 100 testers, aged 18 to 40 years old, in good health, students and teachers, including 40 females and 60 males;
[0049] Step 1.3: Control the ambient noise below 20 decibels. The tester needs to wear the headset in the correct posture, read fluently and clearly, and there should be at least 3 seconds between each sentence to facilitate the subsequent segmentation operation;
[0050] Step 1.4: When the tester is reading, use and to collect the electroglottogram and speech signals respectively, and ensure that there is no noise such as noise generated by friction of the receiving device during recording. Use Goldwave software to record and store the waveform and data of the signal. After the recording is completed, record the tester's gender, age and other information.
[0051] Each step in the entire process of collecting and producing a dual-modal dataset should comply with specific specifications, including tester specifications, data storage specifications, naming specifications, and experimental record specifications. The specific production specifications and their meanings are shown in Table 1.
[0052] Table 1
[0053]
[0054] Tester specifications: 100 testers were selected, aged 18 to 40 years old, students and teachers, including 40 females and 60 males;
[0055] The reason why these 1,000 sentences were selected for the experiment is as follows: 1) to ensure that most of the common phonemes, syllables, tones and grammatical structures in Chinese are covered, so that the dataset can reflect the language characteristics of Chinese as comprehensively as possible; 2) to include different types of vocabulary and sentence structures; 3) the selected sentences can be as balanced and evenly distributed as possible.
[0056] In order to facilitate the organization and retrieval of audio data, a naming rule for bimodal data files is designed. Both audio files and label files must be named according to the naming rule, as shown in Table 2. The file name consists of 11 characters and is divided into four parts.
[0057] Table 2
[0058] file name Tester No. Gender of tester Tester age Test statement number Character length 2 1 2 3 Character Range 00~99 F,M 18~40 000~999
[0059] 1) Tester numbering: The tester information is recorded and numbered. According to the existing tester size (100 people), the tester ID occupies 2 digits, ranging from 00 to 99, and every ten people read the same set of text.
[0060] 2) Gender of the tester: The pitch, formant, intensity and sound quality of the speaker's speech signal vary with the speaker's gender; at the same time, the electroglottogram signal will also differ in glottal period, vocal cord contact ratio and amplitude. The gender identifier occupies 1 character, F: female; M: male.
[0061] 3) Tester age: The speaker's age will affect the speaker's voice in terms of basic frequency, sound quality and formant, and will also affect the speaker's electroglottogram in terms of vocal cord contact pattern, amplitude change and sound stability. The age indicator occupies 2 digits, ranging from 18 to 40.
[0062] 4) Test statement serial number: There are 1000 test statements in total, and each 100 statements are grouped together. Each tester needs to read 100 statements, and statements with the same serial number are the same statement. The statement identifier occupies 3 digits, ranging from 000 to 999.
[0063] The steps to preprocess the dataset are as follows:
[0064] Step 2: Perform denoising, segmentation, naming and other integration operations on the collected data set;
[0065] Step 2.1: After recording, you will get 100 dual-channel .wav files of 30 to 40 minutes in length, with the left channel being speech and the right channel being electroglottogram. Back up this source data.
[0066] Step 2.2: Perform noise reduction on the audio file by using the Empirical Mode Decomposition (EMD) method and process the two channels separately;
[0067] Step 2.3: Segment the left channel according to the pauses, set the segments with sound less than 20 decibels and lasting at least 3 seconds as silent segments, record the time points of each silent segment and audio segment in the entire .wav file, and combine the two channels together for segmentation according to the recorded time points, remove the silent segments and extract the audio segments before and after each silent segment and save them. Each audio file contains 100 texts, and a total of 10,000 .wav files of 7 to 13 seconds are obtained, each containing one text;
[0068] Step 2.4: Name the files in the format of tester number (00-99)_gender (F / M)_age (18-40)_statement number (000-999). The 100 .wav files obtained by the same tester are saved in a folder with the folder name consisting of tester number_gender_age.
[0069] Empirical Mode Decomposition (EMD) is a method for processing nonlinear and non-stationary signals. It helps analyze the essential characteristics of a signal by decomposing a complex signal into several intrinsic mode functions (IMFs) and a residual. EMD is often used for noise reduction because it can effectively separate noise and useful information in a signal.
[0070] The basic principle of using EMD for signal denoising is as follows: First, the original signal is decomposed into several IMFs and a residual through the EMD algorithm. These IMFs have different frequency components, which gradually decrease from high frequency to low frequency, and each IMFs is a single-component function of the local characteristic scale; secondly, because these IMFs mainly include noise, while other IMFs contain useful signal components, by analyzing the frequency characteristics and energy distribution of these IMFs, the IMFs containing the main signal components are selected, among which high-frequency IMFs usually contain more noise, while low-frequency IMFs contain more signal components. Setting a threshold can screen out the IMFs that mainly contain signals; finally, the selected IMFs and the residual are recombined to obtain the denoised signal, and the denoised signal needs to be verified to check whether the main characteristics of the original signal are retained and the noise is effectively removed.
[0071] The advantages of EMD denoising include being able to process nonlinear and non-stationary signals, being adaptive, not requiring preset basis functions, and being able to intuitively separate different frequency components of the signal. However, its limitations are that, first, the computational complexity is high, and it is slow to process long signals; second, it is sensitive to noise, and may need to be combined with other methods (such as threshold processing) for more effective denoising; finally, its results are highly dependent, because different IMF selection strategies will affect the final denoising effect, so the IMF selection threshold is particularly important.
[0072] In general, EMD is a powerful signal processing tool, which is particularly suitable for processing complex nonlinear and non-stationary signals, and is widely used in the fields of noise reduction and feature extraction. Therefore, in this embodiment, EMD is selected to perform noise reduction on the electroglottogram and speech signals. In particular, for the electroglottogram signal, because it has nonlinear and non-stationary characteristics, the traditional linear noise reduction method is not applicable, while EMD can adaptively decompose the signal and effectively process such complex signals.
[0073] The preparation of the mixed speech dataset and the modeling of the target speech extraction task are as follows:
[0074] Step 3: Perform a mixing operation on the original speech and electroglottogram multimodal datasets described in step 2, establish mixed speech and electroglottogram multimodal datasets under different signal-to-noise ratio conditions, and test the target speech extraction model;
[0075] Step 3.1: Randomly select recordings of two speakers from the 100 subjects and mix the two speech segments at a signal-to-noise ratio of 0 dB;
[0076] Step 3.2: Since the length of the original audio varies, in order to facilitate the subsequent recognition processing, the audio is padded and cropped so that the length of all audio is unified to 16 seconds and the sampling rate is 16000;
[0077] Step 3.3: A total of 10,000 mixed audios were produced. The naming method of these audios was a combination of the names of the respective audios before mixing, that is, the form of "target speaker sentence audio file name-interference speaker sentence file name". According to gender, the data set was established into four types of mixed speech data: male-male, female-female, male-female, and female-male. At the same time, the corresponding pure speech and electroglottogram data of the target speaker were also established;
[0078] Step 3.4: Divide the mixed speech and electroglottogram dataset into training and test sets in a ratio of 8:1:1. In order to facilitate management and subsequent data extraction, the detailed information about the mixed audio is recorded in a structured CSV file to facilitate subsequent processing with the corresponding pure speech as the target, train the target speaker speech extraction model based on the convolutional neural network and test it.
[0079] In order to train the target speaker's speech extraction and recognition algorithm, the original clean speech data needs to be mixed to simulate the multi-speaker environment commonly seen in practical applications. Two speakers' recordings are randomly selected from the dual-mode dataset and mixed at a signal-to-noise ratio of 0dB, which means that the sound intensity of the two is adjusted to the same level, so it is impossible to distinguish the main and background speakers by the sound volume alone. This step is intended to increase the difficulty of speech recognition and better simulate the sound conditions under complex conditions. During the dataset production process, the original clean audio was preprocessed. Since the length of the original audio varies, in order to facilitate subsequent recognition processing, the audio is padded and cropped so that the length of all audio is unified to 16 seconds and the sampling rate is 16000Hz. This standardization process not only ensures the consistency of the data, but also improves the efficiency and accuracy of model processing.
[0080] The entire mixing process produced a total of 10,000 mixed audios, which were named in the form of "target speaker sentence audio file name-interference speaker sentence file name" to ensure the traceability of subsequent processing and analysis. All mixed audios were divided into training sets, validation sets, and test sets in a ratio of 8:1:1, which were used for algorithm training, adjustment, and final performance evaluation. In order to facilitate management and subsequent data extraction, detailed information about the mixed audio (i.e., the original audio file names of the target source and interference source involved in the mixing, and the identification of the mixed audio in the new dataset, etc.) were systematically recorded in a structured CSV file. This data management method not only improves the transparency of the experiment, but also facilitates reproduction and further research.
[0081] The purpose of this embodiment is to use a neural network to extract the audio of the target speaker from overlapping mixed speech. Unlike traditional speech separation tasks, this task does not attempt to separate all speech signals in the mixture, but focuses on identifying and extracting a specific target speaker's voice. Therefore, the study adopts the SpeakerBeam method, which makes the model independent of the speaker and universal by utilizing the additional speaker information obtained from the adaptive sentence of the target speaker, because it can extract the target audio by obtaining new speaker adaptation sentences that have not been encountered in the training phase. This additional information can enable the neural network to focus on the target speaker and treat all other sounds as interference, thus effectively avoiding the label arrangement problem in traditional methods.
[0082] This embodiment uses a recurrent neural network (RNN) and a gated recurrent unit (GRU) to train and test the target speech extraction model.
[0083] RNN is a type of neural network used to process sequence data and is widely used in speech recognition, natural language processing and other fields. Unlike traditional feedforward neural networks, RNN has a recurrent connection feature that can capture the temporal dependencies in sequence data. The basic unit of RNN includes an input layer, a hidden layer, and an output layer. The input of each time step depends not only on the current input data, but also on the hidden state of the previous time step.
[0084] GRU is an improved variant of RNN, designed specifically to solve the gradient vanishing problem. GRU introduces a gating mechanism to control the flow of information, making it more suitable for processing long sequence data than standard RNN. The basic structure of GRU includes an update gate and a reset gate. These gates determine how the current hidden state is updated by controlling the flow of information.
[0085] The specific structure of the model is as follows: the speech input signal first processes the audio mixed signal through a one-dimensional convolution layer (encoder layer), converts the original time domain signal into a set of feature representations, and then outputs this set of feature representations to the Bottleneck layer. The main function of the Bottleneck layer is to compress the high-dimensional feature representation to a lower dimension through a linear transformation to reduce the computational complexity. Then in the time domain convolutional network (TCN), the data is processed through multiple convolution blocks and residual connections. Multiple convolution blocks further process these encoded features to capture complex information that is more relevant to the target speaker.
[0086] The auxiliary network is responsible for extracting features from the adaptive speech of the target speaker as an embedding vector. In the experiment, the adaptive speech is the electroglottogram signal corresponding to the speech of the target speaker. After the input electroglottogram signal is converted by the encoder, it is transformed in dimension through the Lambda layer to make it consistent with the input of the time-depth convolutional network. After the feature extraction of the convolution block, the average value is taken in the last dimension to generate a fixed-length embedding vector. Finally, a Lambda layer is used to adjust the dimension of the final speaker embedding vector, and the vector is output to the adaptive layer of the mask network.
[0087] Application of the data set, the modeling of the speech recognition task is as follows:
[0088] Step 4: Create labels and use the dataset to test models such as speech recognition:
[0089] Step 4.1: Label each wav file, that is, a txt file contains the audio reading content and its phonemes, and a total of 10,000 txt label files corresponding to the wav files are obtained;
[0090] Step 4.2: Construct a txt file that provides a list of files and label information for training and testing data to facilitate the model to batch process large amounts of data;
[0091] Step 4.3: To test the practicality and reliability of the dataset, run the dataset using a speech recognition model. The model was built using machine learning and neural network models such as CNN and CTC, and was implemented based on the Keras framework of Tensorflow.
[0092] After the above steps, a dual-mode data set containing 10,000 audio files of speech and electroglottogram information and 10,000 corresponding label files can be obtained. In order to verify the applicability of the data set, the above data results can also be used to establish speech recognition detection through machine learning or neural network models.
[0093] This embodiment intends to use the Convolutional Neutral Network model and the Connectionist Temporal Classification loss function, namely CNN and CTC, to perform speech feature extraction and model training. Through a large amount of data sets and a multi-layer network structure, it can effectively recognize the input of various different speech.
[0094] Unlike RNN, which needs to process data sequentially in time steps, CNN has significant advantages in parallel processing capabilities. The convolution operation of CNN can be performed in parallel on the entire input because each convolution kernel can scan different parts of the input at the same time, which makes CNN faster in training and reasoning, especially when processing images or other structured data. Second, CNN can use local receptive fields to capture local features. By using small-sized convolution kernels (such as 3×3) and multi-layer stacking, CNN can expand the receptive field layer by layer to capture higher-level features, which enables CNN to handle spatial local correlation more effectively and can flexibly control the size of the receptive field by adjusting the convolution kernel size, number of layers, and pooling operations to adapt to different task requirements. Third, the convolution kernels in CNN share parameters on the entire input, which means that the same convolution kernel is used to scan different areas of the input, which can greatly reduce the number of parameters, reduce the complexity of the model and reduce the computational cost. Finally, the hierarchical structure of CNN enables it to extract multi-level features. For example, the initial layer extracts simple edge and texture features. As the number of layers increases, the extracted features become more complex and abstract, which enables CNN to better capture different levels of information in the data.
[0095] CTC is a loss function used for sequence labeling tasks. Unlike other loss functions, it does not require alignment of labels and predicted outputs. In speech recognition tasks, the length of the output sequence usually changes with the input. It can handle label sequences and model output sequences of different lengths in this task because the input and output lengths of speech data may be different and unaligned. Second, its goal is to learn a mapping from the output sequence of the model to the label sequence, rather than learning precise alignment. This correspondence allows the model to learn important features and patterns in the sequence without being restricted by specific time steps. Third, CTC allows the output of empty labels and repeated labels, which makes the model more flexible and adaptable to different types of sequence data. Fourth, it can directly act on the output of the model to achieve end-to-end training, which means that the entire model can be optimized by backpropagation without additional alignment steps or post-processing. CTC is widely used in sequence labeling tasks such as speech recognition, handwriting recognition, and text recognition, and it can achieve good results in these tasks. By minimizing the CTC loss function, the model can gradually optimize its own parameters, thereby improving the ability to accurately recognize input speech features and generate corresponding text labels.
[0096] The specific structure of the model is as follows: the input layer is a 200-dimensional feature value sequence, and the maximum length of a voice data is specified to be 1600 (approximately 16s); the model contains multiple convolutional layers and pooling layers to extract high-level features of the input data. The first hidden layer is a convolutional pooling layer with a convolution kernel size of 3×3 and a pooling window of 2. A batch normalization layer is added after each convolutional layer to accelerate training and improve model stability; the second hidden layer is a fully connected layer; the output layer is a fully connected layer, and softmax is used as the activation function; CTC used CTC's loss as the loss function to achieve connectivity time series multi-output.
[0097] The training process of the adopted model is as follows:
[0098] 1. Determine the batch size and batch size for training, run the CNN model for forward propagation, and learn and extract representative features from the speech data;
[0099] 2. Perform back propagation from output to input, calculate the gradient of the loss function with respect to each weight and bias, and update the weights and biases through the gradient calculated by back propagation to reduce the value of the loss function;
[0100] 3. Use the CTC loss function to learn the mapping relationship from input speech features to output text labels to guide the optimization update of model parameters;
[0101] 4. Repeat the above training and model parameter update process until the number of iterations reaches 50 epochs.
[0102] In this embodiment, a 50-hour database is established, which contains the speech and electroglottogram information of 100 subjects (60 males and 40 females). In training and evaluating speech processing systems, such as speech recognition and speech synthesis, the database can better train and evaluate the model and improve the generalization ability, accuracy, robustness and reliability of the model.
[0103] This embodiment provides a resource of a dual-mode (speech and electroglottogram) data set, which can be used for speaker recognition and voiceprint recognition tasks. By analyzing the characteristics of the two modes, the recognition task can be performed more accurately.
[0104] The dual-mode data set of this embodiment can improve the speech generation and conversion effects. By fusing and analyzing the electroglottogram features and speech features, the emotional state of the speech can be further changed, making the conversion effect more natural and smooth.
[0105] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for generating a multimodal dataset of speech and electroglottomograms, It is characterized in that include: Acquiring original multimodal data read aloud by a plurality of preset persons according to preset text, wherein the original multimodal data includes voice data and electroglottogram data; Performing a mixing operation on each of the original multimodal data to obtain mixed multimodal data under different signal-to-noise ratio conditions; Generate corresponding label information for each of the original multimodal data and each of the mixed multimodal data, store each of the multimodal data and the corresponding label information based on a preset file storage structure, and obtain a multimodal data set.
2. A method for generating a speech and electroglottom multimodal dataset according to claim 1, It is characterized in that Obtaining original multimodal data read aloud by a number of preset people according to preset text, including: Numbering and grouping each preset person and each preset text based on a preset division rule to obtain preset person groups and preset text groups with the same number of groups; According to the grouping order, each preset personnel group and each preset text group are made to correspond one to one, and each group of preset personnel reads the text according to the preset pause interval to obtain a plurality of dual-channel audio files, wherein the dual-channel audio files include a left-channel audio file and a right-channel electroglottogram file; The dual-channel audio files are segmented into text based on a preset pause interval to obtain a plurality of segmented texts, and each segmented text is named to obtain the original multimodal data.
3. The method for generating a speech and electroglottogram multimodal dataset according to claim 2, It is characterized in that In the process of making each group of preset personnel read the text aloud according to the preset pause interval, each of the dual-channel audio files is collected based on a preset audio collection device, and the preset audio collection device includes a microphone and an electroglottogram instrument.
4. The method for generating a speech and electroglottogram multimodal dataset according to claim 2, It is characterized in that Before performing text segmentation on each of the dual-channel audio files based on the preset pause interval, the method also includes performing noise reduction processing on each of the dual-channel audio files based on the empirical mode decomposition method to obtain the noise-reduced dual-channel audio files, and performing text segmentation based on the noise-reduced dual-channel audio files.
5. The method for generating a speech and electroglottogram multimodal dataset according to claim 2, It is characterized in that The text segmentation of each of the two-channel audio files is performed based on a preset pause interval, specifically comprising: An audio segment whose sound is lower than a preset decibel threshold and lasts longer than a preset pause interval is regarded as a silent segment, and an audio segment after removing the silent segment from each dual-channel audio file is extracted as the segmented text.
6. The method for generating a speech and electroglottogram multimodal dataset according to claim 1, It is characterized in that Performing a mixing operation on each of the original multimodal data specifically includes: The original multimodal data of two preset persons are randomly selected, the audio length of the selected original multimodal data is uniformly processed, the audio is mixed according to the set signal-to-noise ratio, and named to obtain the mixed multimodal data.
7. The method for generating a speech and electroglottogram multimodal dataset according to claim 6, It is characterized in that The method further includes storing the mixed multimodal data and the audio information of the corresponding original multimodal data, specifically including: Each of the mixed multimodal data is input into a target speech extraction model for speech extraction to obtain original multimodal data corresponding to each of the mixed multimodal data, and the audio information of each of the mixed multimodal data and the corresponding original multimodal data is stored in a CSV file format; wherein the target speech extraction model is constructed based on a recurrent neural network and a gated recurrent network.
8. The method for generating a speech and electroglottogram multimodal dataset according to claim 1, It is characterized in that The method also includes verifying the availability of the multimodal dataset, specifically including: Each multimodal data in the multimodal data set is input into a speech recognition model for classification and recognition, and predicted label information corresponding to each multimodal data is obtained. The predicted label information is compared with the label information of the corresponding multimodal data to obtain a usability verification result, wherein the speech recognition model is constructed based on a neural network model.
Citation Information
Patent Citations
Method and apparatus for analyzing and synthesizing voice
CN101281744A
Methods for extracting and modeling Chinese speech emotion in combination with glottis excitation and sound channel modulation information
CN102411932A
Voice annotation method for Chinese speech emotion database combined with electroglottography
CN104732981A
EGG (Electroglottograph) voice conversion method based LSTM (Long-short Term Memory)
CN110808026A
A electroglottograph detection device and speech analysis method using EGG and speech signal
KR1020000073638A
