Processing method of large-scale sound effect data set

Through audio length analysis and CLAP model, audio samples are filtered and similarity calculation is performed for text description quality problems, which solves the problem of insufficient quality of large-scale sound effect data sets, improves the similarity between sound effect and text data, and improves model performance .

CN119049512APending Publication Date: 2024-11-29GIANT MOBILE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411247642.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The prior art lacks effective methods for handling large-scale paired sound and text data sets, resulting in low sound quality and text quality and insufficient similarity of paired data, especially in data processing for TTS and ASR. sexual solution.

Method used

By analyzing the audio length distribution, using the CLAP model to filter audio samples, and classifying and filtering the text description quality issues, including general meaningless annotation, abnormal repeated annotation and repeated description annotation, calculating the similarity between audio and text data, and performing Hierarchical sampling and data cleaning, and finally the screened data is used for pre-training and adjustment of the TTA model.

Benefits of technology

The sound quality, text quality and paired text-sound data similarity of large-scale sound effect data sets are improved, providing high-quality training data for large models, thereby improving model performance and application effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049512A_ABST
    Figure CN119049512A_ABST
Patent Text Reader

Abstract

The invention relates to a processing method for a large-scale sound effect data set, and the method comprises the following steps: S1, carrying out the analysis according to the audio length distribution of the data set, and carrying out the processing through a lap model, so as to select an audio sample with the quality meeting the requirement; s2, dividing the text description quality problems into three types, namely universal meaningless annotations, abnormal repeated annotations and repeated description annotations; s3, respectively processing the three types of quality problems, and filtering and deleting corresponding data; and S4, calculating the similarity between each pair of audio and text data based on a lap model, performing stratified sampling evaluation on the data based on similarity distribution, presetting an appropriate second similarity threshold according to an evaluation result, and discarding data pairs smaller than the second similarity threshold. According to the method, the sound effect quality, the text quality and the corresponding similarity degree of the paired text-sound effect data can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for processing a large-scale sound effect data set. Background Art

[0002] The existing processing solutions for large-scale speech datasets mainly focus on TTS (speech synthesis) and ASR (speech recognition) related datasets. For example, TTS has high requirements for the sound quality and naturalness of speech, and the speaker's timbre is uniform in a single data sample; ASR requires accurate corresponding text annotations and strong data generalization. However, there are few data processing solutions for TTA, and they are mainly focused on the generation of text annotations for unlabeled sound effect datasets, such as rule-based manual labeling solutions; or for labeled data, simply labeling it as a sound of + label. These works are either for small data scales or for automatic labeling of audio datasets, but lack processing solutions for improving the quality of large-scale paired sound effect and text datasets.

[0003] Therefore, it is necessary to provide a processing method for a large-scale sound effect dataset to improve the sound effect quality, text quality, and the corresponding similarity of paired text-sound effect data. Summary of the invention

[0004] The object of the present invention is to provide a method for processing a large-scale sound effect data set to improve the sound effect quality, text quality and the corresponding similarity of paired text-sound effect data.

[0005] In order to solve the problems existing in the prior art, the present invention provides a method for processing a large-scale sound effect dataset, comprising the following steps:

[0006] S1: Analyze the audio length distribution of the data set and process it with the help of the clap model to select audio samples with quality that meets the requirements;

[0007] S2: The text description quality issues are divided into three categories: general meaningless annotations, abnormal repeated annotations, and repeated description annotations;

[0008] S3: For the quality problem of general and meaningless annotations, a first similarity scoring threshold is preset, a keyword search is performed on the data with a similarity less than the first similarity scoring threshold, and the searched text description data is filtered and deleted;

[0009] For the quality problem of abnormal repeated annotation, a ratio threshold is preset and the category labeling ratio of the text is calculated. If the category labeling ratio is less than the ratio threshold, the text description data less than the ratio threshold is filtered and deleted;

[0010] For repeated description annotations, the clap model is used to filter and delete the repeated description annotations;

[0011] S4: Calculate the similarity between each pair of audio and text data based on the clap model, perform stratified sampling evaluation on the data based on the similarity distribution, preset an appropriate second similarity threshold according to the evaluation result, and discard data pairs less than the second similarity threshold.

[0012] Optionally, in the method for processing the large-scale sound effect dataset, the way of analyzing the audio length distribution of the dataset is as follows:

[0013] S11: preset the minimum and maximum thresholds of the sound effect length;

[0014] S12: Selecting audio samples whose sound effect length is greater than a minimum threshold and less than a maximum threshold;

[0015] S13: Segment the audio whose sound effect length is greater than the maximum threshold according to the length of the maximum threshold, and select the segmented audio samples.

[0016] Optionally, in the method for processing the large-scale sound effect dataset, the minimum threshold of the sound effect length is 1 second, and the maximum threshold of the sound effect length is 10 seconds.

[0017] Optionally, in the method for processing the large-scale sound effect dataset, the clap model is used for processing to select audio with quality that meets the requirements, in the following manner:

[0018] S14: Preset similarity threshold;

[0019] S15: Calculate the similarity of all adjacent audio sample segments in a single audio sample;

[0020] S16: If the calculated similarity is greater than the similarity threshold, it is considered that different audio sample segments in the audio sample contain similar sound effects, and only the first audio sample segment is retained; if the calculated similarity is less than the similarity threshold, it is considered that different audio sample segments in the audio sample contain different sound effects, and the two adjacent audio sample segments are retained.

[0021] Optionally, in the method for processing the large-scale sound effect dataset, the category labeling ratio of the text is calculated in the following manner: category labeling ratio of the text = non-repeated vocabulary / total vocabulary.

[0022] Optionally, in the method for processing the large-scale sound effect dataset, a clap model is used to filter and delete repeated description annotations in the following manner:

[0023] Preset a second similarity scoring threshold, perform keyword search on data with similarity less than the second similarity scoring threshold, the keywords include and then, followed by, and segment the searched corresponding text to generate two sentences before and after;

[0024] The clap model is used to calculate the similarity between each pair of sentences, and a third similarity scoring threshold is preset. If the similarity calculated according to the clap model is greater than the third similarity scoring threshold, it is considered that the expressions of the previous and next clauses of each pair of sentences are consistent, and only the previous clause is retained in the text description. If it is less than the third similarity scoring threshold, it is considered that the previous and next clauses of each pair of sentences are inconsistent, and the text description retains the previous and next clauses.

[0025] Optionally, the method for processing a large-scale sound effect dataset further includes the following steps:

[0026] S5: Use the unfiltered data for pre-training of the TTA model, and use the filtered data for adjusting the TTA model to further improve model performance.

[0027] In the processing method of the large-scale sound effect data set provided by the present invention, the sound effect quality, text quality and corresponding similarity of paired text-sound effect data in the large model training data set are improved through data analysis and data cleaning, aiming to provide high-quality training data for the large model, thereby improving the performance and application effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flow chart of a method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0030] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0031] The existing processing solutions for large-scale speech datasets mainly focus on TTS (speech synthesis) and ASR (speech recognition) related datasets. For example, TTS has high requirements for the sound quality and naturalness of speech, and the speaker's timbre is uniform in a single data sample; ASR requires accurate corresponding text annotations and strong data generalization. However, there are few data processing solutions for TTA, and they are mainly focused on the generation of text annotations for unlabeled sound effect datasets, such as rule-based manual labeling solutions; or for labeled data, simply labeling it as a sound of + label. These works are either for small data scales or for automatic labeling of audio datasets, but lack processing solutions for improving the quality of large-scale paired sound effect and text datasets.

[0032] In order to solve the problems existing in the prior art, the present invention provides a method for processing a large-scale sound effect dataset, such as Figure 1 As shown, the method comprises the following steps:

[0033] S1: Analyze the audio length distribution of the data set and process it with the help of the clap model to select audio samples with quality that meets the requirements;

[0034] Specifically, the method of analyzing the audio length distribution of the data set is as follows:

[0035] S11: preset the minimum and maximum thresholds of the sound effect length;

[0036] S12: Selecting audio samples whose sound effect length is greater than a minimum threshold and less than a maximum threshold;

[0037] S13: Segment the audio whose sound effect length is greater than the maximum threshold according to the length of the maximum threshold, and select the segmented audio samples.

[0038] Preferably, the minimum threshold of the sound effect length is 1 second, and the maximum threshold of the sound effect length is 10 seconds. For example, in S13, the audio with a sound effect length greater than the maximum threshold is segmented into 10-second segments.

[0039] Furthermore, the clap model is used for processing to select audio with quality that meets the requirements, as follows:

[0040] S14: Preset similarity threshold;

[0041] S15: Calculate the similarity of all adjacent audio sample segments in a single audio sample;

[0042] S16: If the calculated similarity is greater than the similarity threshold, it is considered that different audio sample segments in the audio sample contain similar sound effects, and only the first audio sample segment is retained; if the calculated similarity is less than the similarity threshold, it is considered that different audio sample segments in the audio sample contain different sound effects, and the two adjacent audio sample segments are retained.

[0043] Furthermore, CLAP, the full name of which is "Contrastive Language-Audio Pretraining", is a pre-training method based on contrastive learning of speech and semantic alignment. The goal of this project is to build a powerful and general natural language supervised audio and text encoder pair that can understand sound events and perform high-quality natural language alignment. The CLAP model is trained on a million-level open source text-speech paired dataset, which gives it a wide range of knowledge and context understanding capabilities.

[0044] S2: The text description quality issues are divided into three categories: general meaningless annotations (for example, 'a sound effect is being made'), abnormal repeated annotations (for example, 'Ahi-hat,hi-hat,hi-hat,hi-hat,hi-hat,hi-hat,hi-hat...'), and repeated description annotations (for example, 'A chirping sound is followed byachirping sound').

[0045] S3: (1) To address the quality issues of general and meaningless annotations, a first similarity score threshold is preset, and a keyword search (for example, a keyword such as 'sound effect', because 'soundeffect' means 'sound effect', and any data in the sound effect dataset can be called sound effect, which is equivalent to searching all data) is performed on the data with similarity less than the first similarity score threshold, and the searched text description data is filtered and deleted;

[0046] (2) For the quality problem of abnormal repeated annotation, a ratio threshold is preset, such as 0.6, and the category label ratio (i.e., ttr) of the text is calculated. If the category label ratio is less than the ratio threshold, the text description data less than the ratio threshold is filtered and deleted;

[0047] Further, the method is as follows: the category tag ratio (ttr) of the text = unique words / total words.

[0048] (3) For repeated description annotations, the clap model is used to filter and delete the repeated description annotations;

[0049] Furthermore, the clap model is used to filter and delete repeated description annotations as follows:

[0050] Preset a second similarity scoring threshold, perform keyword search on data with similarity less than the second similarity scoring threshold, the keywords include and then, followed by, and segment the searched corresponding text to generate two sentences before and after;

[0051] The clap model is used to calculate the similarity between each pair of sentences, and a third similarity scoring threshold is preset. If the similarity calculated according to the clap model is greater than the third similarity scoring threshold, it is considered that the expressions of the previous and next clauses of each pair of sentences are consistent, and only the previous clause is retained in the text description. If it is less than the third similarity scoring threshold, it is considered that the previous and next clauses of each pair of sentences are inconsistent, and the text description retains the previous and next clauses.

[0052] In this step, all similarities are calculated as follows:

[0053] Similarity calculation, in the CLAP model, the speech data is input into the speech encoder to generate a one-dimensional speech feature s∈R^d with a length of d, and the text data is input into the text encoder to generate a one-dimensional text feature t∈R^d with a length of d. The similarity α=s·t, where R is a set of real numbers (including all rational and irrational numbers), real number.

[0054] S4: Calculate the similarity between each pair of audio and text data based on the clap model, perform stratified sampling evaluation on the data based on the similarity distribution, preset an appropriate second similarity threshold according to the evaluation result, and discard data pairs less than the second similarity threshold.

[0055] Preferably, the present invention further comprises the following steps:

[0056] S5: Use the unscreened data for pre-training of the TTA (text to audio, TTA) model, and use the screened data for adjusting the TTA model to further improve the model performance. In one embodiment, the data set can be graded according to the actual situation, and the lower-level data (i.e., low-quality data) can be used for pre-training of the TTA model, while the higher-level data (i.e., high-quality data) can be used for fine-tuning the model, which can achieve better results while accelerating convergence.

[0057] In summary, in the processing method of the large-scale sound effect dataset provided by the present invention, the sound effect quality, text quality and corresponding similarity of paired text-sound effect data in the large model training dataset are improved through data analysis and data cleaning, aiming to provide high-quality training data for the large model, thereby improving the performance and application effect of the model.

[0058] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A method for processing a large-scale sound effect dataset, characterized in that: The following steps are involved: S1: Analyze the audio length distribution of the dataset and process it with the help of the clap model to select audio samples with quality that meets the requirements; S2: The text description quality issues are divided into three categories: general meaningless annotations, abnormal repeated annotations, and repeated description annotations; S3: To address the quality issues of general and meaningless annotations, a first similarity score threshold is preset, and a keyword search is performed on data with a similarity less than the first similarity score threshold, and the searched text description data is filtered and deleted; To solve the quality problem of abnormal repeated annotation, a ratio threshold is preset and the category label ratio of the text is calculated. If the category label ratio is less than the ratio threshold, the text description data less than the ratio threshold is filtered out and deleted. For repeated description annotations, the clap model is used to filter and delete the repeated description annotations; S4: Calculate the similarity between each pair of audio and text data based on the clap model, perform stratified sampling evaluation on the data based on the similarity distribution, preset an appropriate second similarity threshold based on the evaluation results, and discard data pairs with a similarity lower than the second similarity threshold.

2. The method for processing a large-scale sound effect dataset according to claim 1, wherein: The analysis based on the audio length distribution of the dataset is as follows: S11: preset minimum and maximum thresholds for the sound effect length; S12: Selecting audio samples whose sound effect length is greater than a minimum threshold and less than a maximum threshold; S13: Segmenting the audio whose sound effect length is greater than the maximum threshold according to the length of the maximum threshold, and selecting the audio samples after segmentation.

3. The method for processing a large-scale sound effect dataset according to claim 2, wherein: The minimum threshold for sound effect length is 1 second, and the maximum threshold for sound effect length is 10 seconds.

4. The method for processing a large-scale sound effect dataset according to claim 2, wherein: Use the clap model to process and select audio with the required quality as follows: S14: Preset similarity threshold; S15: Calculate the similarity of all adjacent audio sample segments in a single audio sample; S16: If the calculated similarity is greater than the similarity threshold, it is considered that different audio sample segments in the audio sample contain similar sound effects, and only the first audio sample segment is retained; if the calculated similarity is less than the similarity threshold, it is considered that different audio sample segments in the audio sample contain different sound effects, and the two adjacent audio sample segments are retained.

5. The method for processing a large-scale sound effect dataset according to claim 1, wherein: The category tag ratio of the text is calculated as follows: category tag ratio of the text = non-repeated words / total words.

6. The method for processing a large-scale sound effect dataset according to claim 1, wherein: The clap model is used to filter and delete duplicate description annotations as follows: Preset a second similarity score threshold, perform keyword search on data with similarity less than the second similarity score threshold, the keywords include and then, followed by, and segment the searched corresponding text to generate two sentences before and after; The clap model is used to calculate the similarity between each pair of sentences, and a third similarity score threshold is preset. If the similarity calculated according to the clap model is greater than the third similarity score threshold, it is considered that the expressions of the previous and next clauses of each pair of sentences are consistent, and only the previous clause is retained in the text description. If it is less than the third similarity score threshold, it is considered that the expressions of the previous and next clauses of each pair of sentences are inconsistent, and the text description retains the previous and next clauses.

7. The method for processing a large-scale sound effect dataset according to claim 1, wherein: The following steps are also included: S5: Use the unfiltered data for pre-training of the TTA model, and use the filtered data for adjusting the TTA model to further improve model performance.