Voice data processing methods, electronic devices, vehicles and storage media

Abstract

This application provides a voice data processing method, electronic device, vehicle, and storage medium. The method includes: acquiring multiple voice data segments and generating corresponding text data; truncating each voice data segment based on a preset duration range to obtain at least one sub-speech data segment and corresponding text data for each sub-speech data segment; extracting acoustic and semantic features of each sub-speech data segment and extracting text vectors from the corresponding text data segment; and storing the acoustic features, semantic features, and text vectors of each sub-speech data segment in a feature database. This application embodiment can truncate voice data based on a preset duration range, reducing the duration differences between different voice data segments, avoiding excessively time-consuming voice feature extraction, and simultaneously extracting effective voice features, effectively improving the quality of voice data and increasing the training efficiency of large-scale voice models.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a voice data processing method, electronic device, vehicle, and storage medium. Background Technology

[0002] In the field of artificial intelligence, large-scale speech models, as a key technology, have received widespread attention. These models enable functions such as speech recognition, speech synthesis, and language understanding, and are widely applied in scenarios such as digital humans, intelligent assistants, and automatic translation. Training large-scale speech models requires a large amount of speech and text data. Since the collected speech data typically comes from different environments or devices, the duration of the data varies considerably. If the speech data is too long, the process of extracting speech features becomes time-consuming; if the speech data is too short, it may be impossible to extract effective speech features, resulting in low training efficiency for large-scale speech models. Summary of the Invention

[0003] In view of the above, it is necessary to provide a speech data processing method, electronic device, vehicle, and storage medium to solve the problem of low training efficiency of large speech models due to the large differences in the duration of speech data.

[0004] In a first aspect, embodiments of this application provide a voice data processing method applied to an electronic device. The method includes: acquiring multiple voice data segments and generating text data corresponding to the voice data; truncating each voice data segment based on a preset duration range to obtain at least one sub-voice data segment corresponding to each voice data segment and text data corresponding to each sub-voice data segment; extracting acoustic features and semantic features of each sub-voice data segment, and extracting text vectors of the text data corresponding to each sub-voice data segment; and storing the acoustic features, semantic features, and text vectors of each sub-voice data segment into a feature database.

[0005] In one possible implementation, the step of truncating each segment of speech data based on a preset duration range to obtain at least one sub-speech data segment and text data corresponding to each sub-speech data segment includes: detecting the main delimiter in the text data corresponding to each segment of speech data based on text order; if the main delimiter is detected and the duration of the first speech data to be truncated corresponding to the main delimiter is within the preset duration range, the first speech data to be truncated is taken as the sub-speech data, and the text data corresponding to the first speech data to be truncated is determined as the text data corresponding to the sub-speech data.

[0006] In one possible implementation, the step of truncating each segment of speech data based on a preset duration range to obtain at least one sub-speech data segment and text data corresponding to each sub-speech data segment includes: detecting the main delimiter in the text data corresponding to each segment of speech data based on the text order; if the main delimiter is detected, and the duration of the first speech data to be truncated corresponding to the main delimiter is less than the preset lower limit duration in the preset duration range, detecting the next main delimiter in the text data corresponding to the speech data.

[0007] In one possible implementation, the step of truncating each segment of speech data based on a preset duration range to obtain at least one sub-speech data segment and text data corresponding to each sub-speech data segment includes: detecting the main delimiter in the text data corresponding to each segment of speech data based on the text order; if the main delimiter is detected, and the duration of the first speech data to be truncated corresponding to the main delimiter is greater than the preset upper limit duration in the preset duration range, detecting the secondary delimiter in the text data corresponding to the first speech data to be truncated.

[0008] In one possible implementation, the step of truncating each segment of speech data based on a preset duration range to obtain at least one sub-segment of speech data corresponding to each segment of speech data includes: setting a preset duration and a preset step size for a sliding window, wherein the preset duration is within the preset duration range and the preset step size is less than or equal to the preset duration; after truncating a sub-segment of speech data from the speech data based on the sliding window, controlling the sliding window to move by the preset step size to truncate the next sub-segment of speech data from the speech data, until the speech data ends.

[0009] In one possible implementation, the method further includes: preprocessing the multiple speech data segments, including: classifying the multiple speech data segments, unifying the format of the multiple speech data segments, and performing noise reduction processing on the multiple speech data segments.

[0010] In one possible implementation, generating the text data corresponding to the speech data includes: dividing the speech data into multiple speech segments based on endpoint detection; and inputting each speech segment into a speech-text recognition model to obtain the corresponding text.

[0011] In one possible implementation, the method further includes: filtering all the cropped sub-speech data to remove sub-speech data containing multiple speakers.

[0012] In one possible implementation, the step of filtering all the cropped sub-speech data to remove sub-speech data containing multiple speakers includes: extracting speech segments from each sub-speech data segment and dividing each speech segment into multiple sub-speech segments; extracting speech features from each sub-speech segment and converting the speech features of each sub-speech segment into speaker embedding vectors using a speaker embedding model; clustering the speaker embedding vectors of all sub-speech segments using a clustering algorithm to obtain the number of clusters, and determining the number of clusters as the number of speakers in the sub-speech data; and filtering out sub-speech data with a speaker count greater than 1.

[0013] In one possible implementation, the method further includes: amplifying each segment of speech data using a timbre conversion model to obtain multiple amplified speech data corresponding to each segment of speech data.

[0014] In one possible implementation, the step of amplifying each segment of sub-speech data using a timbre conversion model to obtain multiple amplified speech data corresponding to each segment of sub-speech data includes: extracting hidden unit features and fundamental frequency features of each segment of sub-speech data using the timbre conversion model; converting the hidden unit features and the fundamental frequency features into first output features respectively; downsampling the first output features corresponding to the hidden unit features and the fundamental frequency features respectively to obtain second output features; fusing the second output features corresponding to the hidden unit features and the fundamental frequency features with the acoustic features of the sub-speech data respectively to obtain fused features corresponding to the hidden unit features and the fundamental frequency features; and upsampling the fused features corresponding to the hidden unit features and the fundamental frequency features respectively to obtain the multiple amplified speech data corresponding to the sub-speech data.

[0015] Thirdly, embodiments of this application provide an electronic device, the electronic device including a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device performs the above-described voice data processing method.

[0016] Fourthly, embodiments of this application provide a vehicle that includes the aforementioned electronic equipment.

[0017] Fifthly, embodiments of this application provide a computer storage medium storing program instructions that, when executed on an electronic device, cause the processor of the electronic device to perform the aforementioned voice data processing method.

[0018] The speech data processing method, electronic device, vehicle, and storage medium provided in this application convert speech data into corresponding text data, extracting text vectors corresponding to the speech data, increasing the diversity and richness of speech data-related features, and truncating speech data based on a preset duration range to reduce and unify the duration of speech data, reduce duration differences between different speech data, avoid excessive time consumption in the speech feature extraction process, facilitate the extraction of effective speech features, and improve the quality of speech sample data and speech features. By pre-storing the extracted speech features in a feature database, the large speech model can directly obtain speech features from the feature database during training, realizing the feature extraction module in advance without needing to obtain speech data, reducing the storage space occupied by speech data, shortening the training time, and effectively improving the training efficiency of the large speech model. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 This is a flowchart of a voice data processing method provided in an embodiment of this application.

[0021] Figure 2 This is a flowchart illustrating the generation of text data corresponding to voice data, provided in one embodiment of this application.

[0022] Figure 3 This is a flowchart of a voice data processing method provided in another embodiment of this application.

[0023] Figure 4 This is a flowchart of a voice data processing method provided in another embodiment of this application.

[0024] Figure 5 This is a flowchart of a voice data processing method provided in another embodiment of this application.

[0025] Figure 6 This is a schematic diagram of the structure of a voice data processing device provided in an embodiment of this application.

[0026] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0027] Figure 8 This is a schematic diagram of the hardware structure of a vehicle provided in one embodiment of this application. Detailed Implementation

[0028] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to limit the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. Where there is no conflict, the following embodiments and features described herein can be combined with each other.

[0030] In the field of artificial intelligence, large-scale speech models, as a key technology, have received widespread attention. These models enable functions such as speech recognition, speech synthesis, and language understanding, and are widely applied in scenarios including digital humans, intelligent assistants, and automatic translation. Training a reliable and efficient large-scale speech model requires a large amount of high-quality speech and text alignment data. In practical applications, acquiring this data faces numerous challenges.

[0031] First, the actual voice data collected often comes from complex sources, including sounds recorded in different environments and with different devices. This voice data is usually accompanied by various background noises, such as traffic noise and crowd noise, which seriously affect the clarity and recognizability of the speech. Furthermore, due to differences in pronunciation habits, speaking speed, and accents among different speakers, voice data recorded in the same environment may also have varying clarity. This leads to the scarcity of high-quality voice data.

[0032] Secondly, due to legal restrictions such as copyright protection and privacy policies, the amount of publicly available speech data is relatively limited. This not only restricts the diversity of speech data but also makes it difficult to meet the requirements of large-scale datasets needed to train large speech models. The insufficient amount of speech data directly affects the generalization ability and accuracy of the model, and is a major bottleneck restricting the development of large speech models.

[0033] In addition, the duration of speech data from different environments or devices varies greatly. If the duration of the speech data is too long, the process of extracting speech features is time-consuming. If the duration of the speech data is too short, it may be impossible to extract effective speech features, resulting in low training efficiency of the large speech model.

[0034] This application provides a speech data processing method that can trim speech data based on a preset duration range, reduce the duration difference between different speech data, avoid excessive time consumption in the speech feature extraction process, and extract effective speech features, thereby effectively improving the quality of speech data and increasing the training efficiency of large speech models.

[0035] See Figure 1 The diagram shown is a flowchart of a voice data processing method provided in an embodiment of this application. The voice data processing method is applied in an electronic device and includes:

[0036] S101: Acquire multiple audio data segments and generate corresponding text data.

[0037] In one embodiment of this application, after obtaining the voice data, endpoint detection is performed on the voice data, the voice data is divided into multiple voice segments based on the endpoints, the voice segments are input into the voice-text recognition model to obtain the text data corresponding to the voice data, the text data is normalized, and the voice data and text data are aligned.

[0038] See Figure 2 The diagram shown is a flowchart of generating text data corresponding to voice data according to an embodiment of this application.

[0039] S1011 divides speech data into multiple speech segments based on endpoint detection.

[0040] In one embodiment of this application, the speech data is framed and windowed, and a short-time Fourier transform is performed on the framed and windowed speech data to obtain the frequency domain features of the speech data. The Mel-frequency cepstral coefficients of the speech data are calculated based on the frequency domain features, including: calculating the square of the amplitude of each frame of speech data to obtain the power spectral density of each frame of speech data; filtering the power spectral density of each frame of speech data through a filter bank to obtain the filtered energy; taking the logarithm of the filtered energy value; performing an inverse Fourier transform on the logarithmic energy spectrum to obtain the cepstral coefficients; and selecting the first preset number of cepstral coefficients (e.g., 12 or 13) as the final Mel-frequency cepstral coefficients.

[0041] In one embodiment of this application, Mel-frequency cepstral coefficients are input into an endpoint detection model. The endpoint detection model outputs multiple endpoints in the speech data. These multiple endpoints include start and end timestamps of multiple speech segments. Based on the start and end timestamps, the speech data is divided into multiple speech segments to filter out non-speech segments. For example, the endpoint detection model is an FSMN (Finite State Machine Network)-VAD (Voice Activity Detection) model. Based on the Mel-frequency cepstral coefficients of the input speech data, the FSMN-VAD model can output a marker sequence. The marker sequence includes multiple endpoints, and these endpoints contain the start and end timestamps of all speech segments in the speech data.

[0042] S1012, input each speech segment into the speech-text recognition model to obtain the corresponding text.

[0043] In one embodiment of this application, each speech segment is extracted and input into a speech-to-text recognition model. The speech features of the speech segments are then extracted using the speech-to-text recognition model. These speech features can be Mel-frequency cepstral coefficients (MFCC) or linear predictive cepstral coefficients (LPCC). The corresponding text is obtained by recognizing the speech features. The speech-to-text recognition model can be a model based on a transformer architecture, including an encoder, decoder, sampler, predictor, etc.

[0044] S1013, normalize the text corresponding to each speech segment.

[0045] In one embodiment of this application, text is converted to uniform lowercase (for English text), redundant spaces or punctuation marks are removed, emojis are removed, and number and date formats are standardized, etc., to reduce the variability of text data.

[0046] S1014, aligns voice data with text data.

[0047] In one embodiment of this application, each speech segment of the speech data is associated with its corresponding text, and the text corresponding to all speech segments constitutes the text data corresponding to the speech data. For example, each speech segment and its corresponding text can be marked with the same identifier, and each speech data segment and its corresponding text data can also be marked with the same identifier to ensure that each speech segment corresponds to a piece of text, and each piece of speech data corresponds to a piece of text.

[0048] S102, each segment of speech data is cut based on a preset duration range to obtain at least one sub-speech data corresponding to each segment of speech data and text data corresponding to each sub-speech data.

[0049] In one embodiment of this application, the preset duration range includes a preset lower limit duration and a preset upper limit duration, for example, the preset lower limit duration is 8 seconds and the preset upper limit duration is 15 seconds. A separator is preset for the text data corresponding to the voice data. The separator can be punctuation marks in the text. The separator includes, but is not limited to, a primary separator and a secondary separator. The primary separator can include punctuation marks used for sentence division such as periods, exclamation marks, and question marks, while the secondary separator can include punctuation marks used for pauses such as commas, semicolons, and colons.

[0050] In one embodiment of this application, the primary delimiter in the text data corresponding to each segment of speech data is detected based on the text sequence, that is, the primary delimiter is detected starting from the beginning position of the text data. If a primary delimiter is detected, the speech data corresponding to the text between the beginning position and the first detected primary delimiter is determined as the first speech data to be truncated. It is then determined whether the duration of the first speech data to be truncated is within a preset duration range. If the duration of the first speech data to be truncated corresponding to the primary delimiter is within the preset duration range, the first speech data to be truncated is used as the sub-speech data, and the text data corresponding to the first speech data to be truncated is determined as the text data corresponding to the sub-speech data. If the duration of the first speech data to be truncated corresponding to the primary delimiter is not within the preset duration range, but is less than a preset lower limit duration, the next primary delimiter in the text data corresponding to the speech data is detected. If the duration of the first speech data to be truncated corresponding to the primary delimiter is not within the preset duration range, but is greater than a preset upper limit duration, the secondary delimiter in the text data corresponding to the first speech data to be truncated is detected.

[0051] In one embodiment of this application, if a secondary separator is detected, it is determined whether the duration of the second speech data to be truncated corresponding to the secondary separator is within a preset duration range. If the duration of the second speech data to be truncated corresponding to the secondary separator is within the preset duration range, the second speech data to be truncated is taken as sub-speech data, and the text data corresponding to the second speech data to be truncated is determined as the text data corresponding to the sub-speech data. If the duration of the second speech data to be truncated corresponding to the secondary separator is not within the preset duration range and is less than a preset lower limit duration, the next secondary separator in the text data corresponding to the first speech data to be truncated is detected.

[0052] In one embodiment of this application, all speech data is cut into at least one segment of speech data by repeating the above method steps.

[0053] In another embodiment of this application, a sliding window is set with a preset duration and a preset step size. This sliding window is used to trim a segment of audio data. The preset duration is within a preset range, and the preset step size is less than or equal to the preset duration. For example, the preset duration is 10 seconds, and the preset step size is 5 seconds. A sub-segment of audio data is trimmed from the beginning of the audio data using the sliding window. After trimming a segment of audio data using the sliding window, the sliding window is controlled to move by the preset step size to trim the next sub-segment of audio data, until the audio data ends. If the duration of the last sub-segment of audio data is less than a preset lower limit, the last segment of audio data is deleted.

[0054] In another embodiment of this application, the above two speech data trimming methods can be used to perform split processing on the trimming of speech data, thereby improving the trimming efficiency of speech data. Alternatively, the above two speech data trimming methods can be used to trim a segment of speech data to obtain two types of sub-speech data corresponding to the segment of speech data, thereby increasing the richness and quantity of speech sample data.

[0055] In one embodiment of this application, after each segment of speech data is cut to obtain at least one corresponding sub-speech data, each sub-speech data and its corresponding text data are aligned.

[0056] S103, extract the acoustic and semantic features of each segment of speech data, and extract the text vector of the corresponding text data for each segment of speech data.

[0057] In one embodiment of this application, each segment of sub-speech data is input into an encoder model, which encodes and compresses the sub-speech data. The encoder model resamples the sub-speech data, for example, converting the sampling rate from 16000Hz to 24000Hz. The resampled sub-speech data is then processed by an encoder and a Residual Vector Quantization (RVQ) to obtain the corresponding discretized vector [B, N, K], thus achieving the compression and encoding of the speech data. Here, B is the batch size, N is the number of audio frames, and K is the number of codebooks, typically 8. This discretized vector is used as the acoustic feature corresponding to the sub-speech data. This discretized vector is a low-dimensional vector, which has high compression efficiency and facilitates data processing during the training process of large speech models. For example, the encoder model is an EnCodec model.

[0058] In one embodiment of this application, the semantic features of the sub-speech data include, but are not limited to, articulation features, intonation features, and emotion features. Each segment of sub-speech data is input into a self-supervised speech representation learning model (HuBERT). The HuBERT model extracts the spectral features of the sub-speech data as articulation features, the prosodic features as intonation features, and the intensity features as emotion features. After extracting the semantic features, the HuBERT model clusters the semantic features using a k-means clustering algorithm, thereby mapping the semantic features to a discrete space.

[0059] In one embodiment of this application, the text data corresponding to each segment of speech data is input into a tokenizer model. The tokenizer model performs word segmentation on the text data, converting the segmented words into text vectors, which are the feature vectors corresponding to the text data. For example, the tokenizer model can be PhonemeBpeTokenizer (a tokenizer model based on phoneme and byte pair encoding).

[0060] S104 stores the acoustic features, semantic features, and text vectors of each segment of speech data into the feature database.

[0061] In one embodiment of this application, the acoustic features, semantic features, and text vectors of each segment of speech data are packaged and stored in a dictionary format, such as JSON (JavaScript Object Notation), XML (eXtensible Markup Language), or YAML (YAML Ain't Markup Language), and stored in the feature database as a file (e.g., a .h5 file). Thus, during the training of the large-scale speech model, the .h5 file can be directly loaded to obtain speech features for training, eliminating the need to load a feature extraction model during training, shortening training time, improving training efficiency, and reducing the memory occupied by training data. In one embodiment of this application, the feature database can be a storage server for storing the training data of the large-scale speech model, the memory of a device used to train the large-scale speech model, or a storage area within the memory of an electronic device for storing the training data of the large-scale speech model.

[0062] Through the above embodiments of this application, speech data can be cropped based on a preset duration range, reducing the duration difference between different speech data, avoiding excessive time consumption in the speech feature extraction process, and extracting effective speech features, thereby effectively improving the quality of speech data and increasing the training efficiency of large speech models.

[0063] See Figure 3 The diagram shown is a flowchart of a voice data processing method provided in another embodiment of this application. The voice data processing method is applied in an electronic device and includes:

[0064] S201: Acquire multiple segments of speech data and preprocess the multiple segments of speech data.

[0065] In one embodiment of this application, preprocessing of multiple speech data segments includes: classifying the multiple speech data segments, unifying the format of the multiple speech data segments, and performing noise reduction processing on the multiple speech data segments.

[0066] In one embodiment of this application, multiple segments of speech data are classified based on multiple preset tags. For example, the preset tags include whether it is purely human voice, whether it contains text, and the language. The preset tag corresponding to whether it is purely human voice includes "pure human voice". <clean>Non-pure human voice <noisy>Whether it contains the preset tag corresponding to the text includes text. <text>Does not contain text<no_text> The preset tags for each language include Chinese. <zh>,English <en>Japanese <jp>Cantonese <yue>and so on. In another embodiment of the present application, whether to include the preset label corresponding to the text can also include aligned text <aligned_text>, ambiguous text <ambiguous_text>, and no text <no_text>. After the multi-segment voice data is classified, the voice data can be processed based on the label.

[0067] In an embodiment of the present application, the format of each segment of voice data is converted into a preset format to unify the format of the multi-segment voice data. The preset format includes but is not limited to a preset sampling rate, a preset number of channels, a preset bit, and a preset file format. For example, the preset sampling rate is 16000 Hz, the preset number of channels is 1 (i.e., single channel), the preset bit is 16 bits, and the preset file format is wav. If the source of the voice data is a video, the audio track of the video can be extracted to obtain the corresponding voice data, and then the format of the voice data is converted into the preset format.

[0068] In an embodiment of the present application, the voice data in the preset format is input into a voice noise reduction model to obtain voice data after noise reduction. The voice noise reduction model performs short-time Fourier transform on the voice data to obtain the spectral features of the voice data, wherein the spectral features are in complex form. The feedforward neural network of the voice noise reduction model analyzes the correlation of the voice signal in the frequency domain and extracts the long sequence features of the spectral features in the time dimension to predict the complex ideal ratio mask (cIRM). The complex ideal ratio mask is multiplied by the original spectrum of the voice signal to obtain the enhanced spectral signal. The enhanced spectral signal is subjected to inverse short-time Fourier transform to obtain the voice data after noise reduction. For example, the voice noise reduction model is a frequency cycle-based CRN (FRCRN) voice noise reduction model.

[0069] S202, generating text data corresponding to the voice data.

[0070] In an embodiment of the present application, for the voice data with the classification label of no text, the corresponding text data is generated. The method of generating the text data corresponding to the voice data is the same as that in S101, which is not described here. For the voice data with the classification label of containing text, the text data corresponding to the voice data can be directly obtained.

[0071] S203, cutting each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data.

[0072] S204, extracting the acoustic features and semantic features of each segment of sub-voice data and extracting the text vector of the text data corresponding to each segment of sub-voice data.

[0073] S205, store the acoustic features, semantic features and text vectors of each sub-voice data to the feature database.

[0074] The specific implementation of S203-S205 is the same as that of S102-S105, and is not described herein.

[0075] Through the above embodiments of the present application, the speech data can be preprocessed by classification, format conversion and noise reduction before feature extraction, effectively improving the quality of the speech data, improving the feature extraction efficiency, and further improving the training efficiency and accuracy of the speech large model.

[0076] Referring to Figure 4 The speech data processing method provided by another embodiment of the present application is shown in the flowchart. The speech data processing method is applied in an electronic device, and the speech data processing method comprises:

[0077] S301, obtain multiple pieces of speech data and preprocess the multiple pieces of speech data.

[0078] S302, generate text data corresponding to the speech data.

[0079] S303, cut each piece of speech data based on a preset time range, to obtain at least one piece of sub-voice data corresponding to each piece of speech data and text data corresponding to each piece of sub-voice data.

[0080] S304, filter all the sub-voice data obtained by cutting, to filter out sub-voice data containing multiple speakers.

[0081] In an embodiment of the present application, a speech segment in each piece of sub-voice data is extracted, each speech segment is divided into multiple sub-segments, speech features of each sub-segment are extracted, the speech features of each sub-segment are converted into speaker embedding vectors by a speaker embedding model, a clustering algorithm is used to cluster the speaker embedding vectors of all the sub-segments, a cluster number is obtained, the cluster number is determined as the number of speakers of the sub-voice data, and the sub-voice data with a speaker number greater than 1 is filtered out.

[0082] Specifically, endpoint detection is performed on each segment of sub-speech data to extract speech segments (i.e., segments containing speech). Each speech segment is divided into multiple sub-speech segments with a preset duration (e.g., 1.5 seconds). Mel-frequency cepstral coefficients of each sub-speech segment are extracted. The Mel-frequency cepstral coefficients of each sub-speech segment are converted into speaker embedding vectors using a speaker embedding model. The k-means clustering algorithm is used to cluster the speaker embedding vectors of all sub-speech segments to obtain the number of clusters. The number of clusters is determined as the number of speakers, thus obtaining the number of speakers in each segment of sub-speech data. Sub-speech data with a number of speakers greater than 1 are filtered out to remove sub-speech data containing multiple speakers.

[0083] In one embodiment of this application, after filtering out sub-speech data containing multiple speakers, the remaining sub-speech data and corresponding text are aligned.

[0084] S305, extract the acoustic and semantic features of each segment of speech data, and extract the text vector of the corresponding text data for each segment of speech data.

[0085] S306 stores the acoustic features, semantic features, and text vectors of each segment of speech data into the feature database.

[0086] The specific implementation methods of S301-S303 and S305-S306 are the same as those of S201-S205, and will not be described in detail here.

[0087] Through the above embodiments of this application, speech data containing multiple speakers can be filtered out, ensuring the consistency of the tone of the speech data and effectively improving the quality of the speech data.

[0088] See Figure 5 The diagram shown is a flowchart of a voice data processing method provided in another embodiment of this application. The voice data processing method is applied in an electronic device and includes:

[0089] S401 acquires multiple audio data segments and preprocesses them.

[0090] S402, Generate text data corresponding to the voice data.

[0091] S403, based on a preset duration range, each segment of speech data is cut to obtain at least one sub-speech data corresponding to each segment of speech data and text data corresponding to each sub-speech data.

[0092] S404 filters all the sub-speech data obtained from the cropping process, removing sub-speech data containing multiple speakers.

[0093] S405 uses a timbre conversion model to amplify each segment of speech data, resulting in multiple amplified speech data corresponding to each segment.

[0094] In one embodiment of this application, the hidden unit HuBERT features and fundamental frequency F0 features of each sub-speech data segment are extracted using a timbre conversion model. The hidden unit features and fundamental frequency features are converted into first output features, respectively. The first output features corresponding to the hidden unit features and fundamental frequency features are downsampled to obtain second output features. The second output features corresponding to the hidden unit features and fundamental frequency features are fused with the acoustic features of the sub-speech data to obtain fused features corresponding to the hidden unit features and fundamental frequency features. The fused features corresponding to the hidden unit features and fundamental frequency features are upsampled to obtain multiple amplified speech data corresponding to the sub-speech data.

[0095] Specifically, the timbre conversion model includes a feature extraction module, a ResNet module, a Unet encoder, and a Unet decoder. Each segment of sub-speech data is input into the timbre conversion model. The feature extraction module extracts HuBERT and F0 features from the sub-speech data. HuBERT features indicate the acoustic characteristics of the sub-speech data, and fundamental frequency features indicate its pitch. The ResNet module converts the HuBERT and fundamental frequency features into first output features of a first preset dimension, where the first preset dimension is (96, T, F), and T represents the time dimension, while F represents the frequency dimension. The first output features corresponding to the HuBERT and fundamental frequency features are input into the Unet encoder. The Unet encoder downsamples the first output features twice in the frequency dimension to obtain second output features of a second preset dimension, where the second preset dimension is (384, T, F / 4). The second output features corresponding to HuberT features and fundamental frequency features are fused with the speaker's speech features (e.g., acoustic features of the sub-speech data) to obtain fused features. These fused features are then input into the Unet decoder. The Unet decoder upsamples the fused features in the frequency dimension to obtain two amplified speech data points corresponding to the sub-speech data. In this embodiment, the second output features corresponding to HuberT features and fundamental frequency features can be upsampled twice by the Unet decoder. Before each upsampling, the second output features are fused with the acoustic features of the sub-speech data to obtain amplified speech data, ensuring that the synthesized amplified speech retains the speaker's acoustic characteristics. In this embodiment, the two amplified speech data points corresponding to the sub-speech data share the same text data as the sub-speech data.

[0096] S406, extract the acoustic and semantic features of each segment of speech data and the amplified speech data, and extract the text vector of the text data corresponding to each segment of speech data and the amplified speech data.

[0097] S407 stores the acoustic features, semantic features, and text vectors of each segment of speech data into the feature database.

[0098] The specific implementation methods of S401-S404 and S406-S407 are the same as those of S301-S306, and will not be described in detail here.

[0099] Through the above embodiments of this application, fictitious data other than real voice data can be constructed, increasing the types and quantity of voice data and improving the generalization ability of large voice models.

[0100] See Figure 6 The diagram shown is a structural schematic of a voice data processing apparatus provided in one embodiment of this application. In one embodiment of this application, the voice data processing apparatus 200 may include multiple functional modules composed of computer program segments. The computer program segments in the voice data processing apparatus 200 may be stored in the memory of an electronic device and executed by at least one processor to perform voice data processing functions.

[0101] In one embodiment of this application, the voice data processing device 200 can be divided into multiple functional modules according to the functions it performs. The functional modules of the voice data processing device 200 may include: a text generation module 201, a voice segmentation module 202, a feature extraction module 203, and a feature storage module 204. In this embodiment, a module refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, and which are stored in memory.

[0102] The text generation module 201 is used to acquire multiple audio data segments and generate text data corresponding to each audio data segment.

[0103] The speech trimming module 202 is used to trim each speech data segment based on a preset duration range to obtain at least one sub-speech data segment corresponding to each speech data segment and text data corresponding to each sub-speech data segment.

[0104] The feature extraction module 203 is used to extract the acoustic and semantic features of each sub-speech data segment, and to extract the text vector of the corresponding text data segment.

[0105] The feature storage module 204 is used to store the acoustic features, semantic features, and text vectors of each segment of speech data into the feature database.

[0106] This application embodiment also provides an electronic device 10, which may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. This application embodiment does not impose any special restrictions on the specific type of electronic device 10.

[0107] See Figure 7 The diagram shown is a hardware structure schematic of an electronic device provided in an embodiment of this application. The voice data processing method provided in this embodiment is applied to an electronic device 10, which includes, but is not limited to, a processor 110 and a memory 120 connected via a communication bus 130. Figure 7 This is merely an example of an electronic device and does not constitute a limitation thereof. In other embodiments, the electronic device may include more components than those shown in the figure.

[0108] The memory 120 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data. The RAM may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0109] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110. Non-volatile memory can include disk storage devices and flash memory.

[0110] The memory 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 110. The one or more computer programs include multiple instructions that, when executed by the processor 110, can implement a voice data processing method that is executed on the electronic device 10.

[0111] In other embodiments, the electronic device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10.

[0112] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0113] The processor 110 provides computing and control capabilities. For example, the processor 110 is used to execute computer programs stored in the memory 120 to implement the above-described voice data processing method.

[0114] The communication bus 130 is used to provide a channel for communication between the memory 120 and the processor 110 in the electronic device 10.

[0115] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0116] This application also provides a vehicle 1, see embodiment 1. Figure 8 The diagram shown is a schematic representation of the hardware structure of a vehicle according to an embodiment of this application. Vehicle 1 includes, as described above... Figure 7 The electronic device 10 shown may be, for example, an in-vehicle device on vehicle 1.

[0117] This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on the electronic device 10, the electronic device 10 performs the aforementioned related method steps to implement the voice data processing method in the above embodiments.

[0118] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the voice data processing method described in the above embodiments.

[0119] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the voice data processing methods in the above-described method embodiments.

[0120] In this application, the electronic devices, computer storage media, computer program products or chips provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0123] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts or all or part of the technical solutions that contribute to the prior art, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.< / yue> < / jp> < / en> < / zh> < / text> < / noisy> < / clean>

Claims

1. A voice data processing method applied to an electronic device, the method comprising: receiving a voice signal; determining a voice type of the voice signal; and processing the voice signal according to the voice type. The method comprises: acquiring multi-section voice data, and generating text data corresponding to the voice data; cutting each section of voice data based on a preset time length range to obtain at least one section of sub-voice data corresponding to each section of voice data and text data corresponding to each section of sub-voice data; extracting acoustic features of each section of sub-voice data; extracting hidden unit features and fundamental frequency features of each section of sub-voice data through a timbre conversion model; converting the hidden unit features and the fundamental frequency features into first output features respectively; performing down-sampling processing on the first output features corresponding to the hidden unit features and the fundamental frequency features respectively to obtain second output features; fusing the second output features corresponding to the hidden unit features and the fundamental frequency features with the acoustic features of each section of sub-voice data to obtain fusion features corresponding to the hidden unit features and the fundamental frequency features; and performing up-sampling processing on the fusion features corresponding to the hidden unit features and the fundamental frequency features to obtain multiple amplified voice data corresponding to each section of sub-voice data; extracting acoustic features and semantic features of the multiple amplified voice data corresponding to each section of sub-voice data, extracting semantic features of each section of sub-voice data, and extracting text vectors of the text data corresponding to each section of sub-voice data and the multiple amplified voice data corresponding to each section of sub-voice data; storing the acoustic features, the semantic features, and the text vectors of each section of sub-voice data and the multiple amplified voice data corresponding to each section of sub-voice data into a feature database.

2. The voice data processing method of claim 1, wherein, The cutting of each section of voice data based on a preset time length range to obtain at least one section of sub-voice data corresponding to each section of voice data and text data corresponding to each section of sub-voice data comprises: detecting a main delimiter in the text data corresponding to each section of voice data in a text sequence; if the main delimiter is detected and the time length of a first to-be-cut voice data corresponding to the main delimiter is within the preset time length range, the first to-be-cut voice data is taken as the sub-voice data, and the text data corresponding to the first to-be-cut voice data is determined as the text data corresponding to the sub-voice data.

3. The voice data processing method of claim 1, wherein, The cutting of each section of voice data based on a preset time length range to obtain at least one section of sub-voice data corresponding to each section of voice data and text data corresponding to each section of sub-voice data comprises: detecting a main delimiter in the text data corresponding to each section of voice data in a text sequence; if the main delimiter is detected and the time length of a first to-be-cut voice data corresponding to the main delimiter is less than a preset lower limit time length in the preset time length range, a next main delimiter in the text data corresponding to the voice data is detected.

4. The voice data processing method of claim 1, wherein, The cutting of each section of voice data based on a preset time length range to obtain at least one section of sub-voice data corresponding to each section of voice data and text data corresponding to each section of sub-voice data comprises: detecting a main delimiter in the text data corresponding to each section of voice data in a text sequence; if the main delimiter is detected and the time length of a first to-be-cut voice data corresponding to the main delimiter is greater than a preset upper limit time length in the preset time length range, a secondary delimiter in the text data corresponding to the first to-be-cut voice data is detected.

5. The voice data processing method of claim 1, wherein, The cutting each piece of voice data based on the preset time length range to obtain at least one piece of sub-voice data corresponding to each piece of voice data comprises: setting a preset time length and a preset step length of a sliding window, the preset time length being within the preset time length range, and the preset step length being less than or equal to the preset time length; after cutting one piece of sub-voice data from the voice data based on the sliding window, controlling the sliding window to move the preset step length to cut the next piece of sub-voice data from the voice data until the voice data ends.

6. The voice data processing method of claim 1, wherein, The method further comprises: preprocessing the multiple pieces of voice data, comprising: classifying the multiple pieces of voice data, unifying the format of the multiple pieces of voice data, and performing noise reduction processing on the multiple pieces of voice data.

7. The voice data processing method of claim 1, wherein, The method further comprises: dividing the voice data into multiple voice segments based on endpoint detection; inputting each voice segment into a voice text recognition model to obtain corresponding text.

8. The voice data processing method of claim 1, wherein, The method further comprises: filtering all the sub-voice data obtained by cutting to filter out sub-voice data containing multiple speakers.

9. The voice data processing method of claim 8, wherein, The filtering all the sub-voice data obtained by cutting to filter out sub-voice data containing multiple speakers comprises: extracting voice segments in each piece of sub-voice data, and dividing each voice segment into multiple sub-segments; extracting voice features of each sub-segment, and converting the voice features of each sub-segment into a speaker embedding vector through a speaker embedding model; performing clustering on the speaker embedding vectors of all sub-segments by using a clustering algorithm to obtain a cluster class number, and determining the cluster class number as the number of speakers of the sub-voice data; filtering out sub-voice data with the number of speakers greater than 1.

10. The voice data processing method of claim 1, wherein, The storing the acoustic features, semantic features and text vectors of each piece of sub-voice data and multiple augmented voice data corresponding to each piece of sub-voice data into a feature database comprises: packaging and storing the acoustic features, semantic features and text vectors of each piece of sub-voice data and multiple augmented voice data corresponding to each piece of sub-voice data into a dictionary format, and storing them into the feature database in the form of a file.

11. An electronic device, comprising: The electronic device comprises a memory and a processor: The memory is configured to store program instructions. The processor is configured to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device executes the voice data processing method according to any one of claims 1 to 10.

12. A vehicle characterized by comprising: The vehicle comprises the electronic device according to claim 11.

13. A computer storage medium, characterized in that The computer storage medium stores program instructions, and when the program instructions are executed on the electronic device, the processor of the electronic device executes the voice data processing method according to any one of claims 1 to 10.

Images