Voice data processing method, electronic equipment, vehicle and storage medium

Abstract

The embodiment of the invention provides a voice data processing method, electronic equipment, a vehicle and a storage medium, and the method comprises the steps: obtaining multiple segments of voice data, and generating text data corresponding to the voice data; cutting each segment of voice data based on a preset duration range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data; extracting acoustic features and semantic features of each segment of sub-voice data, and extracting a text vector of text data corresponding to each segment of sub-voice data; and storing the acoustic features, the semantic features and the text vectors of each segment of sub-voice data into a feature database. According to the embodiment of the invention, the voice data can be cut based on the preset duration range, the duration difference of different voice data is reduced, the time consumption of the voice feature extraction process is avoided, the effective voice features can be extracted, the quality of the voice data is effectively improved, and the training efficiency of the large voice model is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a voice data processing method, electronic equipment, vehicle and storage medium. Background Art

[0002] In the field of artificial intelligence, the development of big speech models as a key technology has received widespread attention. Big speech models can realize functions such as speech recognition, speech synthesis, and language understanding, and are widely used in scenarios such as digital humans, intelligent assistants, and automatic translation. The training of big speech models requires a large amount of speech data and text data. Since the collected speech data usually comes from different environments or different devices, the duration of speech data varies greatly. If the duration of speech data is too long, the process of extracting speech features is more time-consuming. If the duration of speech data is too short, effective speech features may not be extracted, resulting in low training efficiency of big speech models. Summary of the invention

[0003] In view of the above, it is necessary to provide a voice data processing method, electronic device, vehicle and storage medium to solve the problem of low training efficiency of large voice models due to large differences in the duration corresponding to the voice data.

[0004] In a first aspect, an embodiment of the present application provides a voice data processing method, which is applied to an electronic device, and the method includes: acquiring multiple segments of voice data and generating text data corresponding to the voice data; trimming each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data; extracting acoustic features and semantic features of each segment of sub-voice data, and extracting text vectors of text data corresponding to each segment of sub-voice data; storing the acoustic features, semantic features and text vectors of each segment of sub-voice data in a feature database.

[0005] In a possible implementation, each segment of speech data is cut based on a preset duration range to obtain at least one segment of sub-speech data corresponding to each segment of speech data and text data corresponding to each segment of sub-speech data, including: detecting a main separator in the text data corresponding to each segment of speech data based on the text order; if the main separator is detected and the duration of the first speech data to be cut corresponding to the main separator is within the preset duration range, the first speech to be cut is used as the sub-speech data, and the text data corresponding to the first speech data to be cut is determined as the text data corresponding to the sub-speech data.

[0006] In one possible implementation, each segment of speech data is cut based on a preset duration range to obtain at least one segment of sub-speech data corresponding to each segment of speech data and text data corresponding to each segment of sub-speech data, including: detecting a main separator in the text data corresponding to each segment of speech data based on the text order; if the main separator is detected and the duration of the first speech data to be cut corresponding to the main separator is less than a preset lower limit duration in the preset duration range, detecting the next main separator in the text data corresponding to the speech data.

[0007] In one possible implementation, each segment of speech data is cut based on a preset duration range to obtain at least one segment of sub-speech data corresponding to each segment of speech data and text data corresponding to each segment of sub-speech data, including: detecting a main separator in the text data corresponding to each segment of speech data based on the text order; if the main separator is detected and the duration of the first speech data to be cut corresponding to the main separator is greater than a preset upper limit duration in the preset duration range, detecting a secondary separator in the text data corresponding to the first speech data to be cut.

[0008] In a possible implementation, the method of cutting each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data includes: setting a preset time and a preset step of a sliding window, wherein the preset time is within the preset time range, and the preset step is less than or equal to the preset time; after cutting a segment of sub-voice data from the voice data based on the sliding window, controlling the sliding window to move the preset step to cut the next segment of sub-voice data from the voice data until the voice data ends.

[0009] In a possible implementation, the method further includes: preprocessing the multiple segments of voice data, including: classifying the multiple segments of voice data, unifying the formats of the multiple segments of voice data, and performing noise reduction processing on the multiple segments of voice.

[0010] In a possible implementation, generating text data corresponding to the voice data includes: dividing the voice data into multiple voice segments based on endpoint detection; and inputting each voice segment into a voice-text recognition model to obtain corresponding text.

[0011] In a possible implementation manner, the method further includes: filtering all the cut sub-speech data to filter out sub-speech data containing multiple speakers.

[0012] In a possible implementation, filtering all the cut sub-speech data to filter out sub-speech data containing multiple speakers includes: extracting speech segments in each segment of sub-speech data, and dividing each speech segment into multiple sub-segments; extracting speech features of each sub-segment, and converting the speech features of each sub-segment into a speaker embedding vector through a speaker embedding model; clustering the speaker embedding vectors of all sub-segments using a clustering algorithm to obtain the number of clusters, and determining the number of clusters as the number of speakers of the sub-speech data; and filtering out sub-speech data with a speaker number greater than 1.

[0013] In a possible implementation, the method further includes: amplifying each segment of sub-voice data through a timbre conversion model to obtain a plurality of amplified voice data corresponding to each segment of sub-voice data.

[0014] In a possible implementation, the amplification processing of each sub-speech data segment through the timbre conversion model to obtain multiple amplified speech data corresponding to each sub-speech data segment includes: extracting hidden unit features and fundamental frequency features of each sub-speech data segment through the timbre conversion model; converting the hidden unit features and the fundamental frequency features into first output features respectively; down-sampling the first output features corresponding to the hidden unit features and the fundamental frequency features respectively to obtain second output features; fusing the second output features corresponding to the hidden unit features and the fundamental frequency features with the acoustic features of the sub-speech data respectively to obtain fused features corresponding to the hidden unit features and the fundamental frequency features; up-sampling the fused features corresponding to the hidden unit features and the fundamental frequency features respectively to obtain the multiple amplified speech data corresponding to the sub-speech data.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device executes the above-mentioned voice data processing method.

[0016] In a fourth aspect, an embodiment of the present application provides a vehicle, which includes the above-mentioned electronic device.

[0017] In a fifth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores program instructions. When the program instructions are executed on an electronic device, the processor of the electronic device executes the above-mentioned voice data processing method.

[0018] The voice data processing method, electronic device, vehicle and storage medium provided in the embodiments of the present application can convert voice data into corresponding text data, extract text vectors corresponding to the voice data, increase the diversity of voice data-related features and data richness, and trim the voice data based on a preset time range to shorten and unify the time of the voice data, reduce the time difference between different voice data, avoid the time-consuming voice feature extraction process, facilitate the extraction of effective voice features, and improve the quality of voice sample data and voice features. By pre-storing the extracted voice features in a feature database, the large voice model can directly obtain voice features from the feature database during training, realize the front-end feature extraction module, and do not need to obtain voice data, thereby reducing the storage space occupied by voice data and shortening the training time, effectively improving the training efficiency of the large voice model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0020] Figure 1 It is a flowchart of a voice data processing method provided in one embodiment of the present application.

[0021] Figure 2 This is a flowchart of generating text data corresponding to voice data provided by an embodiment of the present application.

[0022] Figure 3 It is a flowchart of a voice data processing method provided by another embodiment of the present application.

[0023] Figure 4 It is a flowchart of a voice data processing method provided by another embodiment of the present application.

[0024] Figure 5 It is a flowchart of a voice data processing method provided by another embodiment of the present application.

[0025] Figure 6 It is a structural diagram of a voice data processing device provided in one embodiment of the present application.

[0026] Figure 7 It is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application.

[0027] Figure 8 It is a schematic diagram of the hardware structure of a vehicle provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms "first" and "second" involved in the embodiments of the present application are only used for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field in this application. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. It should be understood that, unless otherwise specified in this application, " / " means or. For example, A / B can represent A or B. "And / or" in this application is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. "At least one" refers to one or more. "Multiple" refers to two or more than two. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, a, b and c. In the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0030] In the field of artificial intelligence, the development of speech big models as a key technology has received widespread attention. Through speech big models, functions such as speech recognition, speech synthesis, and language understanding can be realized, and they are widely used in scenarios such as digital humans, intelligent assistants, and automatic translation. Training a reliable and efficient speech big model requires a large amount of high-quality speech and text alignment data. In practical applications, there are many challenges in obtaining this data.

[0031] First, the actual collected voice data often comes from complex sources, and may include sounds recorded in different environments and with different devices. These voice data are usually accompanied by various background noises, such as traffic noise, crowd noise, etc., which seriously affect the clarity and recognizability of the voice. In addition, due to differences in pronunciation habits, speaking speed, accents, etc. among different speakers, voice data recorded in the same environment may also have different clarity issues. This leads to the scarcity of high-quality voice data.

[0032] Secondly, due to legal factors such as copyright protection and privacy policy, the amount of publicly available voice data is relatively limited, which not only limits the diversity of voice data, but also makes it difficult to meet the requirements of large-scale data sets required for training large voice models. The lack of voice data directly affects the generalization ability and accuracy of the model, and is the main bottleneck restricting the development of large voice models.

[0033] In addition, the duration of voice data from different environments or different devices varies greatly. If the duration of the voice data is too long, the process of extracting voice features is more time-consuming. If the duration of the voice data is too short, effective voice features may not be extracted, resulting in low training efficiency of large voice models.

[0034] The embodiment of the present application provides a speech data processing method, which can cut speech data based on a preset time length range, reduce the time length difference of different speech data, avoid the time-consuming speech feature extraction process, and at the same time extract effective speech features, effectively improve the quality of speech data, and improve the training efficiency of large speech models.

[0035] See also Figure 1 FIG. 1 is a flow chart of a voice data processing method provided by an embodiment of the present application. The voice data processing method is applied in an electronic device, and the voice data processing method includes: S101, acquiring multiple segments of voice data and generating text data corresponding to the voice data.

[0036] In one embodiment of the present application, after acquiring the voice data, endpoint detection is performed on the voice data, the voice data is divided into multiple voice segments based on the endpoints, the voice segments are input into a voice-text recognition model, text data corresponding to the voice data is obtained, the text data is normalized, and the voice data and text data are aligned.

[0037] See also Figure 2 As shown, it is a flowchart of generating text data corresponding to voice data provided by an embodiment of the present application.

[0038] S1011, dividing the voice data into multiple voice segments based on endpoint detection.

[0039] In one embodiment of the present application, the speech data is framed and windowed, and the short-time Fourier transform is performed on the framed and windowed speech data to obtain the frequency domain features of the speech data. The Mel-cepstral coefficients of the speech data are calculated based on the frequency domain features, including: calculating the square of the amplitude of each frame of speech data to obtain the power spectrum density of each frame of speech data, filtering the power spectrum density of each frame of speech data through a filter group to obtain the filtered energy, taking the logarithm of the filtered energy value, performing an inverse Fourier transform on the logarithmic energy spectrum to obtain the cepstral coefficients, and selecting the cepstral coefficients of the first preset number (for example, 12 or 13) as the final Mel-cepstral coefficients.

[0040] In one embodiment of the present application, the Mel-frequency cepstral coefficients are input into an endpoint detection model, and multiple endpoints in the voice data are output through the endpoint detection model, and the multiple endpoints include the start timestamps and end timestamps of multiple voice segments, and the voice data is divided into multiple voice segments based on the start timestamps and the end timestamps to filter out non-voice segments in the voice data. For example, the endpoint detection model is a FSMN (Finite State Machine Network)-VAD (Voice Activity Detection) model, and the FSMN-VAD model can output a tag sequence based on the Mel-frequency cepstral coefficients of the input voice data, and the tag sequence includes multiple endpoints, and the multiple endpoints include the start timestamps and end timestamps of all voice segments in the voice data.

[0041] S1012, input each speech segment into a speech-to-text recognition model to obtain the corresponding text.

[0042] In one embodiment of the present application, each speech segment is extracted and input into a speech-to-text recognition model, and the speech features of the speech segment are extracted by the speech-to-text recognition model. The speech features may be Mel-frequency cepstral coefficients MFCC or linear prediction cepstral coefficients LPCC, and the speech features are recognized to obtain the corresponding text. The speech-to-text recognition model may be a model based on a transformer architecture, including an encoder, a decoder, a sampler, a predictor, and the like.

[0043] S1013, normalizing the text corresponding to each voice segment.

[0044] In one embodiment of the present application, the text is converted to uniform lowercase (for English text), redundant spaces or punctuation marks are removed, emoticons are removed, and number and date formats are standardized, etc., so as to reduce the variability of text data.

[0045] S1014, aligning the voice data with the text data.

[0046] In one embodiment of the present application, each voice segment of the voice data is associated with the corresponding text, and the texts corresponding to all the voice segments constitute the text data corresponding to the voice data. For example, each voice segment and the corresponding text can be marked with the same identifier, and each segment of voice data and the corresponding text data can also be marked with the same identifier to ensure that each voice segment corresponds to a segment of text, and each segment of voice data corresponds to a segment of text.

[0047] S102, cutting each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data.

[0048] In one embodiment of the present application, the preset duration range includes a preset lower limit duration and a preset upper limit duration, for example, the preset lower limit duration is 8 seconds, and the preset upper limit duration is 15 seconds. A delimiter for text data corresponding to the voice data is preset, and the delimiter may be a punctuation mark in the text. Among them, the delimiter includes but is not limited to: a primary delimiter and a secondary delimiter, the primary delimiter may include a period, an exclamation mark, a question mark, and other punctuation marks used for sentence division, and the secondary delimiter may include a comma, a semicolon, a colon, and other punctuation marks used for pauses.

[0049] In one embodiment of the present application, the main delimiter in the text data corresponding to each segment of voice data is detected based on the text order, that is, the main delimiter is detected from the starting position of the text data. If the main delimiter is detected, the voice data corresponding to the text between the starting position and the first main delimiter detected is determined as the first voice data to be cut, and it is determined whether the duration of the first voice data to be cut is within the preset duration range. If the duration of the first voice data to be cut corresponding to the main delimiter is within the preset duration range, the first voice to be cut is used as the sub-voice data, and the text data corresponding to the first voice data to be cut is determined as the text data corresponding to the sub-voice data. If the duration of the first voice data to be cut corresponding to the main delimiter is not within the preset duration range and is less than the preset lower limit duration, the next main delimiter in the text data corresponding to the voice data is detected. If the duration of the first voice data to be cut corresponding to the main delimiter is not within the preset duration range and is greater than the preset upper limit duration, the secondary delimiter in the text data corresponding to the first voice data to be cut is detected.

[0050] In one embodiment of the present application, if a sub-delimiter is detected, it is determined whether the duration of the second voice data to be cut corresponding to the sub-delimiter is within a preset duration range. If the duration of the second voice data to be cut corresponding to the sub-delimiter is within the preset duration range, the second voice to be cut is used as sub-voice data, and the text data corresponding to the second voice data to be cut is determined as the text data corresponding to the sub-voice data. If the duration of the second voice data to be cut corresponding to the sub-delimiter is not within the preset duration range and is less than the preset lower limit duration, the next sub-delimiter in the text data corresponding to the first voice data to be cut is detected.

[0051] In one embodiment of the present application, all voice data are cut into at least one segment of sub-voice data by looping the above method steps.

[0052] In another embodiment of the present application, a preset duration and a preset step length of a sliding window are set, and the sliding window is used to cut a segment of voice data, the preset duration is within the preset duration range, and the preset step length is less than or equal to the preset duration. For example, the preset duration is 10 seconds, and the preset step length is 5 seconds. A segment of sub-voice data is cut from the starting position of the voice data using a sliding window, and after a segment of sub-voice data is cut from the voice data based on the sliding window, the sliding window is controlled to move the preset step length, and the next segment of sub-voice data is cut from the voice data until the voice data ends. If the duration of the last segment of sub-voice data is less than the preset lower limit duration, the last segment of sub-voice data is deleted.

[0053] In another embodiment of the present application, the above two voice data cutting methods can be used to perform diversion processing on the cutting of voice data, thereby improving the cutting efficiency of voice data. The above two voice data cutting methods can also be used to cut a segment of voice data to obtain two types of sub-voice data corresponding to the segment of voice data, thereby increasing the richness and quantity of voice sample data.

[0054] In one embodiment of the present application, after each segment of voice data is cut to obtain at least one corresponding segment of sub-voice data, each segment of sub-voice data and the corresponding text data are aligned.

[0055] S103, extracting the acoustic features and semantic features of each segment of sub-speech data, and extracting the text vector of the text data corresponding to each segment of sub-speech data.

[0056] In one embodiment of the present application, each segment of sub-speech data is input into an encoder model, and the sub-speech data is encoded and compressed by the encoder model. The encoder model resamples the sub-speech data, for example, converts the sampling rate of 16000Hz to 24000Hz, and processes the resampled sub-speech data through an encoder and a quantizer (Residual Vector Quantization, RVQ) to obtain the corresponding discretized vector [B, N, K], thereby achieving compression and encoding of speech data, where B is the batch size, N is the number of audio frames, and K is the number of codebooks, which is generally 8. The discretized vector is used as the acoustic feature corresponding to the sub-speech data. The discretized vector is a low-dimensional vector with high compression efficiency, which is convenient for data processing of large speech models during training. For example, the encoder model is an EnCodec model.

[0057] In one embodiment of the present application, the semantic features of the sub-speech data include but are not limited to articulation features, intonation features, and emotional features. Each segment of sub-speech data is input into a self-supervised speech representation learning model (HuBERT), and the self-supervised speech representation learning model extracts the spectral features of the sub-speech data as articulation features, extracts the prosodic features of the sub-speech data as intonation features, and extracts the intensity of the sub-speech data as emotional features. After extracting the semantic features, the self-supervised speech representation learning model clusters the semantic features using a k-means clustering algorithm, thereby mapping the semantic features to a discrete space.

[0058] In one embodiment of the present application, the text data corresponding to each sub-speech data is input into a tokenizer model, the text data is tokenized by the tokenizer model, and the vocabulary obtained by the tokenization is converted into a text vector, which is a feature vector corresponding to the text data. For example, the tokenizer model can be PhonemeBpeTokenizer (a tokenizer model based on phoneme and byte pair encoding).

[0059] S104, storing the acoustic features, semantic features and text vector of each sub-speech data segment into a feature database.

[0060] In one embodiment of the present application, the acoustic features, semantic features and text vectors of text data of each segment of sub-speech data are packaged and stored in a dictionary format, for example, the dictionary format is JSON (JavaScript Object Notation), XML (eXtensible Markup Language) or YAML (YAML Ain't Markup Language), and stored in the feature database in the form of a file (such as a .h5 file). In this way, when training a large speech model, the .h5 file can be directly loaded to obtain speech features for training, and there is no need to load the feature extraction model for feature extraction during training, thereby shortening the training time, improving the training efficiency, and reducing the memory occupied by the training data. In one embodiment of the present application, the feature database can be a storage server for storing the training data of the large speech model, can be a memory of a device for training the large speech model, or can be a storage area in the memory of an electronic device for storing the training data of the large speech model.

[0061] Through the above-mentioned embodiments of the present application, the voice data can be trimmed based on a preset time length range, reducing the time length difference between different voice data and avoiding the time-consuming voice feature extraction process. At the same time, effective voice features can be extracted, effectively improving the quality of voice data and improving the training efficiency of large voice models.

[0062] See also Figure 3 FIG. 2 is a flow chart of a voice data processing method provided by another embodiment of the present application. The voice data processing method is applied in an electronic device, and the voice data processing method includes: S201, obtaining multiple segments of voice data, and preprocessing the multiple segments of voice data.

[0063] In one embodiment of the present application, preprocessing the multiple segments of speech data includes: classifying the multiple segments of speech data, unifying the formats of the multiple segments of speech data, and performing noise reduction processing on the multiple segments of speech data.

[0064] In one embodiment of the present application, multiple segments of speech data are classified based on multiple preset tags. For example, the multiple preset tags include whether it is pure human voice, whether it contains text, and language. The preset tags corresponding to whether it is pure human voice include pure human voice. <clean>and non-pure vocals <noisy>, whether it contains text or not, the corresponding preset tags include: <text>, does not contain text<no_text> , the default labels for languages ​​include Chinese <zh>,English <en>, Japanese <jp>, Cantonese <yue>In another embodiment of the present application, whether the preset label corresponding to the text is included may also include the alignment text<aligned_text> , with roughly aligned text<ambiguous_text> , No alignment text<no_text> After multiple segments of speech data are classified, the corresponding processing flow can be performed on the speech data based on the labels.

[0065] In one embodiment of the present application, the format of each segment of voice data is converted into a preset format to unify the formats of multiple segments of voice data. The preset format includes but is not limited to a preset sampling rate, a preset number of channels, a preset bit, and a preset file format. For example, the preset sampling rate is 16000 Hz, the preset number of channels is 1 (i.e., a single channel), the preset bit is 16 bits, and the preset file format is wav. If the source of the voice data is a video, the audio track of the video can be extracted to obtain the corresponding voice data, and then the format of the voice data is converted into the preset format.

[0066] In one embodiment of the present application, speech data in a preset format is input into a speech denoising model to obtain denoised speech data. The speech denoising model performs a short-time Fourier transform on the speech data to obtain spectral features of the speech data, wherein the spectral features are in complex form, and the correlation of the speech signal is analyzed in the frequency domain by the feedforward neural network of the speech denoising model, and the long sequence features of the spectral features are extracted in the time dimension, and a complex ideal ratio mask (cIRM) is predicted, and the complex ideal ratio mask is multiplied by the original spectrum of the speech signal to obtain an enhanced spectral signal, and the enhanced spectral signal is inversely transformed by a short-time Fourier transform to obtain the denoised speech data. For example, the speech denoising model is a speech denoising model based on frequency cyclic CRN (FRCRN).

[0067] S202: Generate text data corresponding to the voice data.

[0068] In one embodiment of the present application, for voice data whose classification label is that it does not contain text, corresponding text data is generated. The method for generating text data corresponding to the voice data is the same as the method in S101, and is not described in detail here. For voice data whose classification label is that it contains text, the text data corresponding to the voice data can be directly obtained.

[0069] S203, cutting each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data.

[0070] S204, extracting the acoustic features and semantic features of each segment of sub-speech data, and extracting the text vector of the text data corresponding to each segment of sub-speech data.

[0071] S205, storing the acoustic features, semantic features and text vector of each sub-speech data segment into a feature database.

[0072] The specific implementation of S203-S205 is the same as that of S102-S105, and will not be repeated here.

[0073] Through the above-mentioned embodiments of the present application, the speech data can be preprocessed by classification, format conversion, and noise reduction before feature extraction, which effectively improves the quality of the speech data, improves the feature extraction efficiency, and thereby improves the training efficiency and accuracy of the large speech model.

[0074] See also Figure 4 FIG. 2 is a flow chart of a voice data processing method provided by another embodiment of the present application. The voice data processing method is applied in an electronic device, and the voice data processing method includes: S301, obtaining multiple segments of voice data, and preprocessing the multiple segments of voice data.

[0075] S302: Generate text data corresponding to the voice data.

[0076] S303, cutting each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data.

[0077] S304: Filter all the cut sub-speech data to remove the sub-speech data containing multiple speakers.

[0078] In one embodiment of the present application, a speech segment is extracted from each sub-speech data, each speech segment is divided into multiple sub-segments, speech features of each sub-segment are extracted, the speech features of each sub-segment are converted into a speaker embedding vector through a speaker embedding model, and the speaker embedding vectors of all sub-segments are clustered using a clustering algorithm to obtain the number of clusters, the number of clusters is determined as the number of speakers of the sub-speech data, and the sub-speech data with a number of speakers greater than 1 is filtered out.

[0079] Specifically, endpoint detection is performed on each segment of sub-speech data, speech segments (i.e., segments containing speech) in the sub-speech data are extracted, each speech segment is divided into multiple sub-segments with a preset duration (e.g., 1.5 seconds), the Mel-frequency cepstral coefficients of each sub-segment are extracted, the Mel-frequency cepstral coefficients of each sub-segment are converted into a speaker embedding vector through a speaker embedding model, the speaker embedding vectors of all sub-segments are clustered using a k-means clustering algorithm to obtain the number of clusters, the number of clusters is determined as the number of speakers, thereby obtaining the number of speakers for each segment of sub-speech data, and the sub-speech data with a number of speakers greater than 1 are filtered out to filter out the sub-speech data containing multiple speakers.

[0080] In one embodiment of the present application, after filtering out sub-speech data containing multiple speakers, the remaining sub-speech data and the corresponding text are aligned.

[0081] S305, extracting the acoustic features and semantic features of each segment of sub-speech data, and extracting the text vector of the text data corresponding to each segment of sub-speech data.

[0082] S306, storing the acoustic features, semantic features and text vector of each sub-speech data segment into a feature database.

[0083] The specific implementations of S301-S303 and S305-S306 are the same as those of S201-S205 and will not be elaborated here.

[0084] Through the above-mentioned embodiments of the present application, voice data containing multiple speakers can be filtered out, the timbre consistency of the voice data can be ensured, and the quality of the voice data can be effectively improved.

[0085] See also Figure 5 FIG. 2 is a flow chart of a voice data processing method provided by another embodiment of the present application. The voice data processing method is applied in an electronic device, and the voice data processing method includes: S401, acquiring multiple voice data segments, and preprocessing the multiple voice data segments.

[0086] S402: Generate text data corresponding to the voice data.

[0087] S403, cutting each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data.

[0088] S404: Filter all the cut sub-speech data to remove the sub-speech data containing multiple speakers.

[0089] S405, amplifying each segment of sub-speech data through a timbre conversion model to obtain a plurality of amplified speech data corresponding to each segment of sub-speech data.

[0090] In one embodiment of the present application, the hidden unit HuBERT features and fundamental frequency F0 features of each segment of sub-speech data are extracted through a timbre conversion model, the hidden unit features and fundamental frequency features are respectively converted into first output features, the first output features corresponding to the hidden unit features and fundamental frequency features are respectively down-sampled to obtain second output features, the second output features corresponding to the hidden unit features and fundamental frequency features are respectively fused with the acoustic features of the sub-speech data to obtain fused features corresponding to the hidden unit features and fundamental frequency features, the fused features corresponding to the hidden unit features and fundamental frequency features are respectively up-sampled to obtain multiple amplified speech data corresponding to the sub-speech data.

[0091] Specifically, the timbre conversion model includes a feature extraction module, a ResNet module, a Unet encoder and a Unet decoder. Each segment of sub-speech data is input into the timbre conversion model, and the HuBERT features and F0 features of the sub-speech data are extracted by the feature extraction module, wherein the HuBERT features are used to indicate the acoustic characteristics of the sub-speech data, and the fundamental frequency features are used to indicate the pitch of the sub-speech data. The HuBERT features and the fundamental frequency features are respectively converted into first output features of a first preset dimension by the ResNet module, wherein the first preset dimension is (96, T, F), wherein T represents the time dimension, and F represents the frequency dimension. The first output features corresponding to the HuBERT features and the fundamental frequency features are respectively input into the Unet encoder, and the first output features are downsampled twice in the frequency dimension by the Unet encoder to obtain the second output features of the second preset dimension, wherein the second preset dimension is (384, T, F / 4). The second output features corresponding to the HuBERT features and the baseband features are respectively fused with the speech features of the speaker (e.g., the acoustic features of the sub-speech data) to obtain fused features, and the fused features corresponding to the HuBERT features and the baseband features are respectively input into the Unet decoder, and the fused features corresponding to the HuBERT features and the baseband features are respectively upsampled in the frequency dimension by the Unet decoder to obtain two amplified speech data corresponding to the sub-speech data. In an embodiment of the present application, the second output features corresponding to the HuBERT features and the baseband features can be upsampled twice by the Unet decoder, and before each upsampling, the second output features are fused with the acoustic features of the sub-speech data to obtain amplified speech data, so that the synthesized amplified speech retains the acoustic characteristics of the speaker. In an embodiment of the present application, the two amplified speech data corresponding to the sub-speech data share the same text data as the sub-speech data.

[0092] S406, extracting the acoustic features and semantic features of each segment of sub-speech data and augmented speech data, and extracting the text vector of the text data corresponding to each segment of sub-speech data and augmented speech data.

[0093] S407, storing the acoustic features, semantic features and text vector of each sub-speech data in a feature database.

[0094] The specific implementations of S401-S404 and S406-S407 are the same as those of S301-S306, and are not elaborated here.

[0095] Through the above-mentioned embodiments of the present application, fictitious data other than real voice data can be constructed, the types and quantity of voice data can be increased, and the generalization ability of the large voice model can be improved.

[0096] See also Figure 6 , which is a schematic diagram of the structure of a voice data processing device provided in one embodiment of the present application. In one embodiment of the present application, the voice data processing device 200 may include a plurality of functional modules composed of computer program segments. The computer program segments in the voice data processing device 200 may be stored in a memory of an electronic device and executed by at least one processor to perform the voice data processing function.

[0097] In one embodiment of the present application, the speech data processing device 200 can be divided into multiple functional modules according to the functions it performs. The functional modules of the speech data processing device 200 may include: a text generation module 201, a speech cutting module 202, a feature extraction module 203, and a feature storage module 204. The module in the embodiment of the present application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory.

[0098] The text generation module 201 is used to obtain multiple voice data segments and generate text data corresponding to each voice data segment.

[0099] The speech cutting module 202 is used to cut each segment of speech data based on a preset time range to obtain at least one segment of sub-speech data corresponding to each segment of speech data and text data corresponding to each segment of sub-speech data.

[0100] The feature extraction module 203 is used to extract the acoustic features and semantic features of each segment of sub-speech data, and to extract the text vector of the text data corresponding to each segment of sub-speech data.

[0101] The feature storage module 204 is used to store the acoustic features, semantic features and text vectors of each segment of sub-speech data into a feature database.

[0102] The embodiment of the present application also provides an electronic device 10, which can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device and / or a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the electronic device 10.

[0103] See also Figure 7 The figure is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The voice data processing method provided in the embodiment of the present application is applied to an electronic device 10, which includes, but is not limited to, a processor 110 and a memory 120 connected via a communication bus 130. Figure 7 This is merely an example of an electronic device and does not constitute a corresponding limitation. In other embodiments, the electronic device may include more components than those shown in the figure.

[0104] The memory 120 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0105] The non-volatile memory may also store executable programs and user and application data, etc., and may be pre-loaded into the random access memory for direct reading and writing by the processor 110. The non-volatile memory may include a disk storage device and a flash memory.

[0106] The memory 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 110. The one or more computer programs include a plurality of instructions, and when the plurality of instructions are executed by the processor 110, the voice data processing method executed on the electronic device 10 can be implemented.

[0107] In other embodiments, the electronic device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0108] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0109] The processor 110 provides computing and control capabilities. For example, the processor 110 is used to execute a computer program stored in the memory 120 to implement the above-mentioned voice data processing method.

[0110] The communication bus 130 is at least used to provide a channel for the memory 120 and the processor 110 in the electronic device 10 to communicate with each other.

[0111] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0112] The present application also provides a vehicle 1, referring to Figure 8 FIG. 1 is a schematic diagram of the hardware structure of a vehicle provided in an embodiment of the present application. The vehicle 1 includes: Figure 7 The electronic device 10 shown, for example, may be an on-board device on the vehicle 1 .

[0113] The embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on the electronic device 10, the electronic device 10 executes the above-mentioned related method steps to implement the voice data processing method in the above-mentioned embodiment.

[0114] The embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, the computer executes the above-mentioned related steps to implement the voice data processing method in the above-mentioned embodiment.

[0115] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the chip executes the voice data processing method in the above-mentioned method embodiments.

[0116] Among them, the electronic device, computer storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0117] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0118] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program code.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.< / yue> < / jp> < / en> < / zh> < / text> < / noisy> < / clean>

Claims

1. A voice data processing method, applied to electronic equipment, characterized in that: The method comprises: Acquire multiple segments of voice data and generate text data corresponding to the voice data; Cut each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data; Extracting acoustic features and semantic features of each segment of sub-speech data, and extracting text vectors of text data corresponding to each segment of sub-speech data; The acoustic features, semantic features and text vectors of each sub-speech data are stored in the feature database.

2. The method for processing speech data according to claim 1, wherein: The method of cutting each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data includes: Detecting the main separator in the text data corresponding to each segment of speech data based on the text order; If the main separator is detected and the duration of the first voice data to be cut corresponding to the main separator is within the preset duration range, the first voice to be cut is used as the sub-voice data, and the text data corresponding to the first voice data to be cut is determined as the text data corresponding to the sub-voice data.

3. The method for processing speech data according to claim 1, wherein: The method of cutting each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data includes: Detecting the main separator in the text data corresponding to each segment of speech data based on the text order; If the main separator is detected, and the duration of the first voice data to be cut corresponding to the main separator is less than the preset lower limit duration in the preset duration range, the next main separator in the text data corresponding to the voice data is detected.

4. The method for processing speech data according to claim 1, wherein: The method of cutting each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data and text data corresponding to each segment of sub-voice data includes: Detecting the main separator in the text data corresponding to each segment of speech data based on the text order; If the main separator is detected, and the duration of the first voice data to be cut corresponding to the main separator is greater than the preset upper limit duration in the preset duration range, a secondary separator in the text data corresponding to the first voice data to be cut is detected.

5. The method for processing speech data according to claim 1, wherein: The step of cutting each segment of voice data based on a preset time range to obtain at least one segment of sub-voice data corresponding to each segment of voice data includes: Setting a preset duration and a preset step length of the sliding window, wherein the preset duration is within the preset duration range, and the preset step length is less than or equal to the preset duration; After a segment of sub-speech data is cut from the speech data based on the sliding window, the sliding window is controlled to move the preset step length to cut the next segment of sub-speech data from the speech data until the speech data ends.

6. The method for processing speech data according to claim 1, wherein: The method further comprises: Preprocessing the multiple segments of speech data includes: The multiple segments of speech data are classified, the formats of the multiple segments of speech data are unified, and noise reduction processing is performed on the multiple segments of speech data.

7. The method for processing speech data according to claim 1, wherein: The generating text data corresponding to the voice data comprises: Dividing the speech data into a plurality of speech segments based on endpoint detection; Input each speech segment into the speech-to-text recognition model to obtain the corresponding text.

8. The method for processing speech data according to claim 1, wherein: The method further comprises: All the cut sub-speech data are filtered to remove the sub-speech data containing multiple speakers.

9. The method for processing speech data according to claim 8, wherein: The filtering process of all the cut sub-speech data to remove the sub-speech data containing multiple speakers includes: Extracting a speech segment from each segment of sub-speech data, and dividing each speech segment into a plurality of sub-segments; Extract the speech features of each sub-segment and convert the speech features of each sub-segment into a speaker embedding vector through a speaker embedding model; Clustering the speaker embedding vectors of all sub-segments using a clustering algorithm to obtain a cluster number, and determining the cluster number as the number of speakers of the sub-speech data; The sub-speech data with the number of speakers being greater than 1 are filtered out.

10. The method for processing speech data according to claim 1, wherein: The method further comprises: Each segment of sub-speech data is amplified by using a timbre conversion model to obtain a plurality of amplified speech data corresponding to each segment of sub-speech data.

11. The method for processing speech data according to claim 10, characterized in that: The amplification process of each sub-speech data segment is performed by using the timbre conversion model to obtain a plurality of amplified speech data corresponding to each sub-speech data segment, including: Extracting hidden unit features and fundamental frequency features of each segment of sub-speech data through the timbre conversion model; Converting the hidden unit feature and the base frequency feature into a first output feature respectively; Downsampling the first output features corresponding to the hidden unit features and the baseband features respectively to obtain second output features; Respectively fusing the second output features corresponding to the hidden unit feature and the fundamental frequency feature with the acoustic features of the sub-speech data to obtain fused features corresponding to the hidden unit feature and the fundamental frequency feature; The hidden unit feature and the fusion feature corresponding to the fundamental frequency feature are up-sampled respectively to obtain the multiple amplified speech data corresponding to the sub-speech data.

12. The method for processing speech data according to claim 1, wherein: The step of storing the acoustic features, semantic features and text vectors of each sub-speech data in a feature database includes: The acoustic features, the semantic features and the text vector of each segment of sub-speech data are packaged and stored in a dictionary format, and stored in the feature database in the form of a file.

13. An electronic device, characterized in that: The electronic device comprises a memory and a processor: Wherein, the memory is used to store program instructions; The processor is used to read and execute the program instructions stored in the memory. When the program instructions are executed by the processor, the electronic device executes the voice data processing method as described in any one of claims 1 to 12.

14. A vehicle, characterized in that: The vehicle includes the electronic device as claimed in claim 13.

15. A computer storage medium, characterized in that The computer storage medium stores program instructions, and when the program instructions are executed on an electronic device, a processor of the electronic device executes the voice data processing method according to any one of claims 1 to 12.

Images