Audio data processing method, device and electronic equipment

Abstract

The embodiments of the present application provide an audio data processing method, device and electronic device, which can achieve: transcribing each audio data to be processed into text data, and grouping the text data; deduplicating each group of text data, and selecting a target audio data that meets the preset saturation requirement from each audio data corresponding to each group of text data after deduplication, and adding it to the training data set; when the number of audio data in the training data set is greater than or equal to the preset threshold, output the training data set; otherwise, training a voiceprint model according to the audio data in the training data set, extracting a voiceprint vector using the trained voiceprint model, and selecting available audio data from the candidate audio data set according to the voiceprint vector and adding it to the training data set. The embodiments of the present application can filter out audio data with high saturation and diversified text from massive audio data, and improve the model performance of the voiceprint recognition model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of voiceprint recognition technology, and in particular to an audio data processing method, device and electronic device. Background Art

[0002] Voiceprint recognition is a type of biometric technology, also known as speaker recognition, which includes speaker identification and speaker confirmation.

[0003] In the prior art, the training of text-independent voiceprint recognition models based on deep learning relies on massive amounts of audio data. In the process of audio data collection, for some devices, the collected audio data often has the following characteristics: (1) There is silence and noise before and after the audio, and the audio saturation is low; (2) The transcribed texts corresponding to the audio may be highly similar, which is not conducive to maintaining the diversity of text and the diversity of pronunciation contained in the audio; (3) There is no speaker label corresponding to the audio, and manual annotation may be required.

[0004] However, manually screening and labeling massive amounts of audio data would be a huge workload, and blindly exporting massive amounts of audio data would take up a lot of memory space. Therefore, how to effectively screen out audio data with high saturation and diverse text from massive amounts of audio data is a technical problem that urgently needs to be solved. Summary of the invention

[0005] The embodiments of the present application provide an audio data processing method, device and electronic device, which can effectively filter out audio data with high saturation and text diversity from massive audio data, thereby improving the model performance of the voiceprint recognition model.

[0006] In a first aspect, an embodiment of the present application provides an audio data processing method, the method comprising:

[0007] Transcribing at least one piece of audio data to be processed into text data, and grouping the text data;

[0008] Deduplication processing is performed on each group of text data, and a target audio data satisfying a preset saturation requirement is selected from each audio data corresponding to each group of text data after deduplication processing and added to a preset training data set;

[0009] Determining whether the amount of audio data in the training data set is less than a preset threshold;

[0010] When the number of audio data in the training data set is not less than the preset threshold, the training data set is used as a target training data set for the voiceprint model to be trained;

[0011] When the number of audio data in the training data set is less than the preset threshold, the voiceprint model is trained according to the audio data in the training data set, a voiceprint vector is extracted using the trained voiceprint model, and available audio data is selected from the candidate audio data set according to the voiceprint vector, and the available audio data is added to the training data set; returning to the step of determining whether the number of audio data in the training data set is less than the preset threshold, until the number of audio data in the training data set is not less than the preset threshold, the training data set is used as the target training data set for the voiceprint model to be trained;

[0012] The candidate audio data set includes the remaining audio data to be processed in each set of text data after deduplication processing except the audio data added to the training data set.

[0013] In a feasible implementation manner, extracting a voiceprint vector according to the trained voiceprint model, and selecting available audio data from a candidate audio data set according to the voiceprint vector, includes:

[0014] Determine the voiceprint vector of each piece of audio data in the training data set based on the trained voiceprint model, and calculate the average voiceprint vector of each piece of audio data corresponding to the same electronic device in the training data set according to the voiceprint vector of each piece of audio data in the training data set;

[0015] Determine a voiceprint vector of each piece of audio data in the candidate audio data set;

[0016] Calculating the similarity between the voiceprint vector of each piece of audio data corresponding to the same electronic device in the candidate audio data set and the average voiceprint vector;

[0017] Determine a target voiceprint vector having the greatest similarity to the average voiceprint vector among the voiceprint vectors of the audio data corresponding to the same electronic device in the candidate audio data set;

[0018] The audio data corresponding to the target voiceprint vector is used as the available audio data.

[0019] In a feasible implementation manner, after adding the available audio data to the training data set, the method further includes:

[0020] The available audio data is deleted from the candidate audio data set.

[0021] In a feasible implementation manner, grouping the text data includes:

[0022] The text data are grouped based on an identity ID of an electronic device that collects the audio data to be processed, wherein the audio data to be processed corresponding to each group of text data is collected by the same electronic device.

[0023] In a feasible implementation manner, the deduplication processing of each set of text data includes:

[0024] For each set of text data, the following processing is performed:

[0025] Add the first text in the current group of text data to the text library corresponding to the current group of text data;

[0026] Traversing each text in the current set of text data, and calculating the similarity between the currently traversed text and each text in the text library;

[0027] If the similarity between the currently traversed text and each text in the text library is less than a preset similarity threshold, the currently traversed text is added to the text library;

[0028] If the similarity between the currently traversed text and one of the texts in the text library is greater than or equal to the similarity threshold, the currently traversed text is discarded.

[0029] In a feasible implementation manner, calculating the similarity between the currently traversed text and each text in the text library includes:

[0030] The word error rate CER of the currently traversed text and the target text to be calculated in the text library is calculated in the following manner:

[0031]

[0032] The similarity COR between the currently traversed text and the target text to be calculated in the text library is calculated in the following way:

[0033] COR=1-CER

[0034] Among them, S represents the word replacement difference value between the currently traversed text and the target text to be calculated in the text library, D represents the word deletion difference value between the currently traversed text and the target text to be calculated in the text library, I represents the word insertion difference value between the currently traversed text and the target text to be calculated in the text library, and N represents the total number of characters of the target text to be calculated in the text library.

[0035] In a feasible implementation manner, selecting a target audio data satisfying a preset saturation requirement from each audio data corresponding to each group of text data after deduplication processing and adding it to a preset training data set includes:

[0036] Removing the silent segments in the audio data corresponding to each group of text data after the deduplication processing, to obtain the first audio data corresponding to each group of text data after the deduplication processing;

[0037] Among the first audio data corresponding to each group of text data after the deduplication processing, audio data whose file size is greater than a preset byte threshold and whose number of transcribed text words is greater than a preset word count threshold are selected as the second audio data corresponding to each group of text data after the deduplication processing;

[0038] Calculate the saturation of each piece of second audio data corresponding to each set of text data after the deduplication processing according to the file size of each piece of second audio data corresponding to each set of text data after the deduplication processing and the number of words in the transcribed text;

[0039] The second audio data with the largest saturation among the second audio data corresponding to each group of text data after the deduplication processing is added to the training data set.

[0040] In a feasible implementation manner, after adding the second audio data with the largest saturation among the second audio data corresponding to each group of text data after the deduplication processing to the training data set, the method further includes:

[0041] Among the pieces of second audio data corresponding to each group of text data after the deduplication process, the remaining second audio data except the second audio data with the highest saturation are added to a preset candidate audio data set.

[0042] In a second aspect, an embodiment of the present application provides an audio data processing device, the device comprising:

[0043] A grouping module, used for transcribing at least one piece of audio data to be processed into text data, and grouping the text data;

[0044] A screening module is used to perform deduplication processing on each set of text data, and select a target audio data that meets the preset saturation requirement from each audio data corresponding to each set of text data after deduplication processing, and add it to a preset training data set;

[0045] A determination module, used to determine whether the amount of audio data in the training data set is less than a preset threshold;

[0046] A processing module, configured to use the training data set as a target training data set for a voiceprint model to be trained when the number of audio data in the training data set is not less than the preset threshold;

[0047] When the number of audio data in the training data set is less than the preset threshold, the voiceprint model is trained according to the audio data in the training data set, a voiceprint vector is extracted according to the trained voiceprint model, and available audio data is selected from the candidate audio data set according to the voiceprint vector, and the available audio data is added to the training data set; returning to the step of determining whether the number of audio data in the training data set is less than the preset threshold, until the number of audio data in the training data set is not less than the preset threshold, the training data set is used as the target training data set for the voiceprint model to be trained;

[0048] The candidate audio data set includes the remaining audio data to be processed in each set of text data after deduplication processing except the audio data added to the training data set.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, including: at least one processor and a memory;

[0050] The memory stores computer-executable instructions;

[0051] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the audio data processing method provided in the first aspect.

[0052] The audio data processing method, device and electronic device provided in the embodiments of the present application can achieve: transcribing at least one audio data to be processed into text data, and grouping the text data; deduplicating each group of text data, and selecting a target audio data that meets the preset saturation requirement from each audio data corresponding to each group of text data after deduplication, and adding it to the training data set; when the number of audio data in the training data set is not less than the preset threshold, output the training data set; otherwise, train the voiceprint model according to the audio data in the training data set, extract the voiceprint vector using the trained voiceprint model, and select available audio data from the alternative audio data set according to the voiceprint vector and add it to the training data set, until the number of audio data in the training data set is not less than the above-mentioned preset threshold. The embodiments of the present application can filter out audio data with high saturation and text diversity from massive audio data by the method of screening data-training model-screening data-training model, thereby improving the model performance of the voiceprint recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 A flowchart of an audio data processing method provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of a sub-process of an audio data processing method provided in an embodiment of the present application;

[0056] Figure 3 Another schematic diagram of a flow chart of an audio data processing method provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a program module of an audio data processing device provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. In addition, although the disclosure in the present application is introduced according to one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation method separately.

[0060] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0061] The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this application.

[0062] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0063] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0064] Voiceprint recognition is a type of biometric technology, also known as speaker recognition, which includes speaker identification and speaker confirmation. It converts sound signals into electrical signals and then uses computers to identify them.

[0065] The so-called voiceprint is a sound wave spectrum that carries speech information displayed by an electroacoustic instrument. The production of human language is a complex physiological and physical process between the human body's language center and the vocal organs. The vocal organs used by people when speaking - tongue, teeth, larynx, lungs, and nasal cavity - vary greatly in size and shape from person to person, so the voiceprints of any two people are different. The acoustic characteristics of each person's voice are both relatively stable and variable, and are not absolute and unchanging. This variation can come from physiology, pathology, psychology, simulation, disguise, and is also related to environmental interference. Despite this, since everyone's vocal organs are different, in general, people can still distinguish the voices of different people or judge whether they are the same person's voice.

[0066] Compared with other biometric features, the application of voiceprint recognition has some special advantages: (1) Voice containing voiceprint features is easy and natural to obtain, and voiceprint extraction can be completed unconsciously, so users are highly accepting of it; (2) The cost of obtaining voice recognition is low and easy to use, with only a microphone. When using communication equipment, no additional recording equipment is required; (3) It is suitable for remote identity confirmation. Only a microphone or a telephone or a mobile phone is needed to log in remotely through the network (communication network or Internet); (4) The algorithm complexity of voiceprint recognition and confirmation is low; (5) In combination with some other measures, such as content identification through voice recognition, the accuracy can be improved; ... and so on.

[0067] Different tasks and applications use different voiceprint recognition technologies, namely speaker identification and speaker verification. The former is used to determine which of several people said a certain speech, which is a "multiple-choice" problem; while the latter is used to confirm whether a certain speech is said by a specified person, which is a "one-to-one discrimination" problem. Different tasks and applications use different voiceprint recognition technologies. Whether it is identification or verification, the speaker's voiceprint needs to be modeled first, which is the so-called "training" or "learning" process.

[0068] There are two types of voiceprint recognition: text-dependent and text-independent. The text-dependent voiceprint recognition system requires users to pronounce according to the specified content. Each person's voiceprint model is accurately established one by one, and the pronunciation must also be in accordance with the specified content during recognition. Therefore, it can achieve a better recognition effect, but the system requires user cooperation. If the user's pronunciation does not match the specified content, the user cannot be correctly identified. The text-independent recognition system does not specify the speaker's pronunciation content, and the model establishment is relatively difficult, but it is easy for users to use and has a wide range of applications.

[0069] The training of text-independent voiceprint recognition models based on deep learning relies on massive speaker audio data. However, in real life, collecting audio from massive speakers requires a lot of time and manpower. Therefore, automatic speaker matching based on a large amount of acquired audio can greatly reduce the time and manpower required for labeling, screening or recording audio.

[0070] Taking television as an example, the audio data collected by the TV end often has the following characteristics: (1) the number of speakers contained in each TV ID generally does not exceed 5, and there is no speaker label; (2) there is a lot of audio data; (3) the corresponding transcription text of the audio is similar; (4) the speakers of different TVs generally have no intersection. Therefore, a reasonably designed voiceprint data screening method can provide important data guarantee for voiceprint recognition.

[0071] The disadvantages of TV audio data compared to recorded audio are: (1) The audio is not full, and there is silence and noise before and after the audio; (2) There are situations where the audio and the corresponding transcribed text are highly similar, which is not conducive to maintaining text diversity and the diversity of pronunciation contained in the audio; (3) There are no speaker labels corresponding to the audio, and manual labeling may be required. However, manually screening and labeling tens of millions of audio data is a huge project, and blindly exporting audio data requires a large amount of memory space. In addition, it requires a lot of manpower. Therefore, how to automatically screen full and text-diverse audio data is a technical problem that needs to be solved urgently.

[0072] In the face of the above technical problems, the embodiments of the present application provide an audio data processing method, which, by means of screening data-training model-screening data-training model, enriches the training data of the training set while continuously improving the model performance of the voiceprint recognition model. In addition, the memory space required for the audio can be greatly reduced by deduplicating and screening the text and then exporting the audio. And the audio data processing method provided in the embodiments of the present application is not dependent on model selection, so it can be applied to various voiceprint recognition deep learning models, and therefore has a certain degree of universality. The following detailed embodiments are used for detailed explanation.

[0073] Reference Figure 1 , Figure 1 A flowchart of the audio data processing method provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0074] S101: transcribe at least one piece of audio data to be processed into text data, and group each piece of text data.

[0075] In some embodiments, multiple electronic devices may collect multiple audios as audio data to be processed, and then after transcribing each audio data to be processed into text data, each text data may be grouped according to the attributes of each audio data to be processed.

[0076] Optionally, the electronic device may be a television, a mobile terminal, a computer or other device, which is not limited in the embodiments of the present application.

[0077] S102, performing deduplication processing on each group of text data, and selecting a target audio data satisfying a preset saturation requirement from each audio data corresponding to each group of text data after deduplication processing, and adding the target audio data to a preset training data set.

[0078] In some embodiments, deduplication processing is performed on each set of text data to remove highly similar texts in each set of text data, thereby obtaining texts with diversity.

[0079] In addition, one target audio data with the highest saturation can be selected from each audio data corresponding to each group of text data after deduplication processing and added to a preset training data set, and the remaining audio data can be added to a preset candidate audio data set.

[0080] S103, determining whether the amount of audio data in the training data set is less than a preset threshold; if not, continuing to execute S104; if yes, executing S105.

[0081] S104: Use the above training data set as the target training data set for the voiceprint model to be trained.

[0082] In some embodiments, when the number of audio data in the above-mentioned training data set is greater than or equal to the above-mentioned preset threshold, it can be considered that the audio data in the above-mentioned training data set has met the voiceprint model's requirements for input audio data. Therefore, the above-mentioned training data set can be directly used as the target training data set for the voiceprint model to be trained.

[0083] S105, training the voiceprint model according to the audio data in the training data set, extracting voiceprint vectors using the trained voiceprint model, and selecting available audio data from the candidate audio data set according to the voiceprint vectors, and adding the available audio data to the training data set. Return to step S103.

[0084] The candidate audio data set includes the remaining audio data in each set of text data after deduplication processing except the audio data added to the training data set.

[0085] In some embodiments, when the number of audio data in the above-mentioned training data set is less than the above-mentioned preset threshold, it can be considered that the audio data in the above-mentioned training data set cannot meet the voiceprint model's requirements for input audio data. Therefore, it is necessary to train the above-mentioned voiceprint model based on the audio data in the above-mentioned training data set, and then use the trained voiceprint model to extract the voiceprint vector of the audio data in the above-mentioned training data set, and select available audio data from the alternative audio data set based on the voiceprint vector and add it to the above-mentioned training data set to expand the above-mentioned training data set.

[0086] When the number of expanded audio data in the above training data set is greater than or equal to the above preset threshold, it can be considered that the audio data in the above training data set has met the voiceprint model's requirements for input audio data. Therefore, the expanded training data set can be used as the target training data set for the voiceprint model to be trained.

[0087] The audio data processing method provided in the embodiment of the present application can filter out audio data with high saturation and text diversity from massive audio data by means of filtering data-training model-filtering data-training model, which helps to improve the model performance of the voiceprint recognition model.

[0088] Based on the contents described in the above embodiments, refer to Figure 2 , Figure 2 A schematic diagram of a sub-process of the audio data processing method provided in an embodiment of the present application, such as Figure 2 As shown, in the above step S105, the voiceprint vector is extracted by using the trained voiceprint model, and available audio data is selected from the candidate audio data set according to the voiceprint vector, including:

[0089] S201. Determine the voiceprint vector of each piece of audio data in the training data set based on the trained voiceprint model, and calculate the average voiceprint vector of each piece of audio data corresponding to the same electronic device in the training data set according to the voiceprint vector of each piece of audio data in the training data set.

[0090] It can be understood that the audio data in the above training data set is audio data with high saturation and text diversity screened out from the audio data to be processed. Therefore, based on the above average voiceprint vector, it can be determined what kind of audio data with voiceprint vector is needed for the same electronic device to train the voiceprint model.

[0091] S202: Determine the voiceprint vector of each piece of audio data in the candidate audio data set.

[0092] S203: Calculate the similarity between the voiceprint vector of each audio data corresponding to the same electronic device in the candidate audio data set and the average voiceprint vector.

[0093] S204: Determine a target voiceprint vector having the greatest similarity to the average voiceprint vector among the voiceprint vectors of the audio data corresponding to the same electronic device in the candidate audio data set.

[0094] S205. Use the audio data corresponding to the target voiceprint vector as the above-mentioned available audio data.

[0095] In an embodiment of the present application, after determining what kind of voiceprint vector audio data is needed for the same electronic device to train the voiceprint model, as well as the voiceprint vectors of each audio data in the alternative audio data set, the target voiceprint vector with the greatest similarity to the above-mentioned average voiceprint vector is selected from the voiceprint vectors of each audio data corresponding to the same electronic device in the alternative audio data set, and then the audio data corresponding to the target voiceprint vector is used as the above-mentioned available audio data.

[0096] In order to better understand the embodiments of the present application, refer to Figure 3 , Figure 3 Another flowchart of the audio data processing method provided in the embodiment of the present application is as follows: Figure 3 As shown, the above audio data processing method includes:

[0097] (1) Transcribe each piece of audio data to be processed into text data.

[0098] (2) Text grouping:

[0099] In a feasible implementation, the text data may be grouped based on the identity ID of the electronic device that collects each piece of audio data to be processed.

[0100] The audio data to be processed corresponding to each group of text data after grouping are collected by the same electronic device.

[0101] (3) Text deduplication:

[0102] In a feasible implementation, the following processing is performed for each set of text data:

[0103] Add the first text in the current group of text data to the text library corresponding to the current group of text data.

[0104] Traverse each text in the current set of text data, and calculate the similarity between the currently traversed text and each text in the text library.

[0105] If the similarity between the currently traversed text and each text in the above text library is less than a preset similarity threshold, the currently traversed text is added to the text library.

[0106] If the similarity between the currently traversed text and one of the texts in the text library is greater than or equal to the above similarity threshold, the currently traversed text is discarded.

[0107] In a feasible implementation, the word error rate CER of the currently traversed text and the target text to be calculated in the text library can be calculated in the following manner:

[0108]

[0109] Then the similarity COR between the currently traversed text and the target text to be calculated in the text library is calculated in the following way:

[0110] COR=1-CER

[0111] Among them, S represents the word replacement difference value between the currently traversed text and the target text to be calculated in the text library, D represents the word deletion difference value between the currently traversed text and the target text to be calculated in the text library, I represents the word insertion difference value between the currently traversed text and the target text to be calculated in the text library, and N represents the total number of characters of the target text to be calculated in the text library.

[0112] (4) Extract audio data and remove silent segments.

[0113] In a feasible implementation, each audio data corresponding to each group of text data after deduplication processing can be obtained first, and then the silent segments in each audio data corresponding to each group of text data are removed to obtain each first audio data corresponding to each group of text data after deduplication processing.

[0114] Optionally, the corresponding audio file in the database is extracted based on the audio ID corresponding to the text, and then the front and back silent segments of the audio file are removed based on the energy.

[0115] Among the first audio data corresponding to each group of text data after deduplication processing, the audio data whose file size is greater than a preset byte threshold and whose number of transcribed text words is greater than a preset word count threshold are selected as the second audio data corresponding to each group of text data after deduplication processing.

[0116] Exemplarily, audio data that satisfies both the audio size being greater than 150 kb and the number of transcribed text characters being greater than 6 may be screened out as the second audio data.

[0117] The saturation of each piece of second audio data corresponding to each group of text data after the deduplication processing is calculated based on the file size of each piece of second audio data corresponding to each group of text data after the deduplication processing and the number of words in the transcribed text.

[0118] Optionally, the above saturation = number of text words / file size of audio data.

[0119] The second audio data with the largest saturation among the second audio data corresponding to each group of text data after deduplication processing is added to the training data set.

[0120] Among the pieces of second audio data corresponding to each group of text data after deduplication processing, the remaining second audio data except the second audio data with the highest saturation is added to the preset candidate audio data set.

[0121] (5) Determine whether the amount of audio data in the training data set is less than a preset threshold; if so, continue to execute (6); if not, output the training data set.

[0122] (6) Train the voiceprint model.

[0123] In a feasible implementation, the voiceprint model may be trained based on the audio data in the training data set.

[0124] (7) Extract the voiceprint vector and take its mean.

[0125] In a feasible implementation, the voiceprint vector of each piece of audio data in the training data set can be determined based on the trained voiceprint model, and the average voiceprint vector of each piece of audio data corresponding to the same electronic device in the training data set can be calculated based on the voiceprint vector of each piece of audio data in the training data set. That is, the same electronic device ultimately corresponds to one voiceprint vector.

[0126] (8) Extract the voiceprint vector of each audio data in the alternative audio data set.

[0127] In a feasible implementation, the voiceprint vector of each audio data in the candidate audio data set may be determined based on the trained voiceprint model.

[0128] (9) Similarity scoring.

[0129] In a feasible implementation, the similarity between the voiceprint vectors of each audio data corresponding to the same electronic device in the candidate audio data set and the average voiceprint vector is calculated.

[0130] (10) Select available audio data.

[0131] In a feasible implementation, a target voiceprint vector having the greatest similarity to the voiceprint vector among the voiceprint vectors of the audio data corresponding to the same electronic device in the candidate audio data set is determined; and the audio data corresponding to the target voiceprint vector is used as the available audio data.

[0132] (11) Add available audio data to the above training dataset.

[0133] In a feasible implementation, after adding the available audio data to the training data set, continue to execute (5).

[0134] (12) Delete the above available audio data from the candidate audio data set.

[0135] The audio data processing method provided in the embodiment of the present application can achieve: transcribing each piece of audio data to be processed into text data, and grouping the text data; deduplicating each group of text data, and selecting a target audio data that meets the preset saturation requirement from each piece of audio data corresponding to each group of text data after deduplication, and adding it to the training data set; when the number of audio data in the training data set is greater than or equal to the preset threshold, output the training data set; otherwise, training the voiceprint model according to the audio data in the training data set, extracting the voiceprint vector according to the trained voiceprint model, and selecting available audio data from the candidate audio data according to the voiceprint vector and adding it to the training data set. The embodiment of the present application can filter out audio data with high saturation and text diversity from massive audio data through the method of screening data-training model-screening data-training model, thereby improving the model performance of the voiceprint recognition model.

[0136] Based on the contents described in the above embodiments, an audio data processing device is also provided in the embodiments of the present application. Figure 4 , Figure 4 This is a schematic diagram of a program module of an audio data processing device provided in an embodiment of the present application, the audio data processing device comprising:

[0137] The grouping module 401 is used to transcribe each piece of audio data to be processed into text data and group the text data.

[0138] The screening module 402 is used to perform deduplication processing on each group of text data, and select a target audio data that meets the preset saturation requirement from each audio data corresponding to each group of text data after deduplication processing and add it to the preset training data set.

[0139] The judgment module 403 is used to judge whether the amount of audio data in the training data set is less than a preset threshold.

[0140] The processing module 404 is used to use the training data set as a target training data set for the voiceprint model to be trained when the number of audio data in the training data set is greater than or equal to the preset threshold.

[0141] When the number of audio data in the training data set is less than the preset threshold, the voiceprint model is trained according to the audio data in the training data set, the voiceprint vector is extracted using the trained voiceprint model, and available audio data is selected from the alternative audio data set according to the voiceprint vector, and the available audio data is added to the training data set, and the step of determining whether the number of audio data in the training data set is less than the preset threshold is returned to execute, wherein the alternative audio data set is the remaining audio data in each group of text data after deduplication processing except the audio data added to the training data set.

[0142] The audio data processing device provided in the embodiment of the present application can filter out audio data with high saturation and text diversity from massive audio data through the method of filtering data-training model-filtering data-training model, thereby improving the model performance of the voiceprint recognition model.

[0143] In a feasible implementation manner, the grouping module 401 is used to:

[0144] The text data are grouped based on the identity ID of the electronic device that collects each piece of audio data to be processed, wherein the audio data to be processed corresponding to each group of text data is collected by the same electronic device.

[0145] In a feasible implementation, the screening module 402 is used to:

[0146] For each set of text data, the following processing is performed:

[0147] The first text in the current group of text data is added to the text library corresponding to the current group of text data; each text in the current group of text data is traversed, and the similarity between the currently traversed text and each text in the text library is calculated; if the similarity between the currently traversed text and each text in the text library is less than a preset similarity threshold, the currently traversed text is added to the text library; if the similarity between the currently traversed text and one of the texts in the text library is greater than or equal to the similarity threshold, the currently traversed text is discarded.

[0148] In a feasible implementation, the screening module 402 is specifically used for:

[0149] The word error rate CER of the currently traversed text and the target text to be calculated in the text library is calculated in the following manner:

[0150]

[0151] The similarity COR between the currently traversed text and the target text to be calculated in the text library is calculated in the following way:

[0152] COR=1-CER

[0153] Among them, S represents the word replacement difference value between the currently traversed text and the target text to be calculated in the text library, D represents the word deletion difference value between the currently traversed text and the target text to be calculated in the text library, I represents the word insertion difference value between the currently traversed text and the target text to be calculated in the text library, and N represents the total number of characters of the target text to be calculated in the text library.

[0154] In a feasible implementation, the screening module 402 is further used to:

[0155] Remove the silent segments in each audio data corresponding to each group of text data after deduplication processing to obtain each first audio data corresponding to each group of text data after deduplication processing; among the first audio data corresponding to each group of text data after deduplication processing, select the audio data whose file size is greater than a preset byte threshold and whose number of transcribed text words is greater than a preset word count threshold as each second audio data corresponding to each group of text data after deduplication processing; calculate the saturation of each second audio data corresponding to each group of text data after deduplication processing according to the file size of each second audio data corresponding to each group of text data after deduplication processing and the number of transcribed text words; add the second audio data with the largest saturation among the second audio data corresponding to each group of text data after deduplication processing to the training data set.

[0156] In a feasible implementation, the screening module 402 is further used to:

[0157] Among the pieces of second audio data corresponding to each group of text data after deduplication processing, the remaining second audio data except the second audio data with the highest saturation is added to the preset candidate audio data set.

[0158] In a feasible implementation manner, the processing module 404 is further configured to:

[0159] Based on the trained voiceprint model, determine the voiceprint vector of each audio data in the training data set, and calculate the average voiceprint vector of each audio data corresponding to the same electronic device in the training data set based on the voiceprint vector of each audio data in the training data set; determine the voiceprint vector of each audio data in the alternative audio data set; calculate the similarity between the voiceprint vector of each audio data corresponding to the same electronic device in the alternative audio data set and the average voiceprint vector; determine the target voiceprint vector whose similarity to the average voiceprint vector is the largest among the voiceprint vectors of each audio data corresponding to the same electronic device in the alternative audio data set; and use the audio data corresponding to the target voiceprint vector as the available audio data.

[0160] In a feasible implementation manner, the processing module 404 is further configured to:

[0161] After adding the above available audio data to the training data set, the available audio data is deleted from the candidate audio data set.

[0162] It should be noted that the specific contents of the grouping module 401, the screening module 402, the judging module 403 and the processing module 404 in the embodiment of the present application can be referred to in Figures 1 to 3 The relevant contents in the illustrated embodiment will not be described in detail here.

[0163] Furthermore, based on the contents described in the above embodiments, an electronic device is also provided in the embodiments of the present application, which includes at least one processor and a memory; wherein the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory to implement the various steps of the audio data processing method described in the above embodiments. For details, please refer to the contents described in the above embodiments, which will not be repeated in this embodiment.

[0164] In order to better understand the embodiments of the present application, refer to Figure 5 , Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0165] like Figure 5 As shown, the electronic device 50 of this embodiment includes: a processor 501 and a memory 502; wherein:

[0166] Memory 502, used to store computer-executable instructions;

[0167] The processor 501 is used to execute the computer-executable instructions stored in the memory to implement the various steps of the audio data processing method described in the above embodiment. For details, please refer to the contents described in the above embodiment, which will not be repeated in this embodiment.

[0168] Optionally, the memory 502 may be independent or integrated with the processor 501 .

[0169] When the memory 502 is independently provided, the device further includes a bus 503 for connecting the memory 502 and the processor 501 .

[0170] Furthermore, based on the contents described in the above embodiments, a computer-readable storage medium is also provided in the embodiments of the present application, in which computer execution instructions are stored. When the processor executes the computer execution instructions, the various steps of the audio data processing method described in the above embodiments are implemented. For details, please refer to the contents described in the above embodiments, and this embodiment will not be repeated here.

[0171] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0172] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, each functional module in each embodiment of the present application can be integrated into a processing unit, each module can exist physically separately, or two or more modules can be integrated into one unit. The above module integration unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0174] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application.

[0175] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0176] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0177] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0178] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0179] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0180] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio data processing method, characterized in that: The method comprises: Transcribing at least one piece of audio data to be processed into text data, and grouping the text data; Deduplication processing is performed on each group of text data, and a target audio data satisfying a preset saturation requirement is selected from each audio data corresponding to each group of text data after deduplication processing and added to a preset training data set; Determining whether the amount of audio data in the training data set is less than a preset threshold; When the number of audio data in the training data set is not less than the preset threshold, the training data set is used as a target training data set for the voiceprint model to be trained; When the number of audio data in the training data set is less than the preset threshold, the voiceprint model is trained according to the audio data in the training data set, a voiceprint vector is extracted using the trained voiceprint model, and available audio data is selected from the candidate audio data set according to the voiceprint vector, and the available audio data is added to the training data set; returning to the step of determining whether the number of audio data in the training data set is less than the preset threshold, until the number of audio data in the training data set is not less than the preset threshold, the training data set is used as the target training data set for the voiceprint model to be trained; The candidate audio data set includes the remaining audio data to be processed in each set of text data after deduplication processing except the audio data added to the training data set; The step of selecting a target audio data satisfying a preset saturation requirement from each audio data corresponding to each group of text data after deduplication processing and adding the target audio data to a preset training data set includes: Removing the silent segments in the audio data corresponding to each group of text data after the deduplication processing, to obtain the first audio data corresponding to each group of text data after the deduplication processing; Among the first audio data corresponding to each group of text data after the deduplication processing, audio data whose file size is greater than a preset byte threshold and whose number of transcribed text words is greater than a preset word count threshold are selected as the second audio data corresponding to each group of text data after the deduplication processing; Calculate the saturation of each piece of second audio data corresponding to each set of text data after the deduplication processing according to the file size of each piece of second audio data corresponding to each set of text data after the deduplication processing and the number of words in the transcribed text; The second audio data with the largest saturation among the second audio data corresponding to each group of text data after the deduplication processing is added to the training data set.

2. The method according to claim 1, characterized in that The extracting voiceprint vectors by using the trained voiceprint model, and selecting available audio data from the candidate audio data set according to the voiceprint vectors, includes: Determine the voiceprint vector of each piece of audio data in the training data set based on the trained voiceprint model, and calculate the average voiceprint vector of each piece of audio data corresponding to the same electronic device in the training data set according to the voiceprint vector of each piece of audio data in the training data set; Determine a voiceprint vector of each piece of audio data in the candidate audio data set; Calculating the similarity between the voiceprint vector of each piece of audio data corresponding to the same electronic device in the candidate audio data set and the average voiceprint vector; Determine a target voiceprint vector having the greatest similarity to the average voiceprint vector among the voiceprint vectors of the audio data corresponding to the same electronic device in the candidate audio data set; The audio data corresponding to the target voiceprint vector is used as the available audio data.

3. The method according to claim 1 or 2, characterized in that: After adding the available audio data to the training data set, the method further includes: The available audio data is deleted from the candidate audio data set.

4. The method according to claim 1, characterized in that: The grouping of the text data comprises: The text data are grouped based on an identity ID of an electronic device that collects the audio data to be processed, wherein the audio data to be processed corresponding to each group of text data is collected by the same electronic device.

5. The method according to claim 1, characterized in that: The deduplication processing of each set of text data includes: For each set of text data, the following processing is performed: Add the first text in the current group of text data to the text library corresponding to the current group of text data; Traversing each text in the current set of text data, and calculating the similarity between the currently traversed text and each text in the text library; If the similarity between the currently traversed text and each text in the text library is less than a preset similarity threshold, the currently traversed text is added to the text library; If the similarity between the currently traversed text and one of the texts in the text library is greater than or equal to the similarity threshold, the currently traversed text is discarded.

6. The method according to claim 5, characterized in that Calculate the similarity between the currently traversed text and each text in the text library, including: The word error rate CER of the currently traversed text and the target text to be calculated in the text library is calculated in the following manner: The similarity COR between the currently traversed text and the target text to be calculated in the text library is calculated in the following way: Among them, S represents the word replacement difference value between the currently traversed text and the target text to be calculated in the text library, D represents the word deletion difference value between the currently traversed text and the target text to be calculated in the text library, I represents the word insertion difference value between the currently traversed text and the target text to be calculated in the text library, and N represents the total number of characters of the target text to be calculated in the text library.

7. The method according to claim 1, characterized in that After adding the second audio data with the largest saturation among the second audio data corresponding to each group of text data after the deduplication process to the training data set, the method further includes: Among the pieces of second audio data corresponding to each group of text data after the deduplication process, the remaining second audio data except the second audio data with the highest saturation are added to a preset candidate audio data set.

8. An audio data processing device, characterized in that: include: A grouping module, used for transcribing at least one piece of audio data to be processed into text data, and grouping the text data; A screening module is used to perform deduplication processing on each set of text data, and select a target audio data that meets the preset saturation requirement from each audio data corresponding to each set of text data after deduplication processing, and add it to a preset training data set; A determination module, used to determine whether the amount of audio data in the training data set is less than a preset threshold; A processing module, configured to use the training data set as a target training data set for a voiceprint model to be trained when the number of audio data in the training data set is not less than the preset threshold; When the number of audio data in the training data set is less than the preset threshold, training the voiceprint model according to the audio data in the training data set, extracting a voiceprint vector using the trained voiceprint model, and selecting available audio data from the candidate audio data set according to the voiceprint vector, and adding the available audio data to the training data set; Return to the step of determining whether the number of audio data in the training data set is less than a preset threshold, until the number of audio data in the training data set is not less than the preset threshold, and use the training data set as a target training data set for the voiceprint model to be trained; The candidate audio data set includes the remaining audio data to be processed in each set of text data after deduplication processing except the audio data added to the training data set; The step of selecting a target audio data satisfying a preset saturation requirement from each audio data corresponding to each group of text data after deduplication processing and adding the target audio data to a preset training data set includes: Removing the silent segments in the audio data corresponding to each group of text data after the deduplication processing, to obtain the first audio data corresponding to each group of text data after the deduplication processing; Among the first audio data corresponding to each group of text data after the deduplication processing, audio data whose file size is greater than a preset byte threshold and whose number of transcribed text words is greater than a preset word count threshold are selected as the second audio data corresponding to each group of text data after the deduplication processing; Calculate the saturation of each piece of second audio data corresponding to each set of text data after the deduplication processing according to the file size of each piece of second audio data corresponding to each set of text data after the deduplication processing and the number of words in the transcribed text; The second audio data with the largest saturation among the second audio data corresponding to each group of text data after the deduplication processing is added to the training data set.

9. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the audio data processing method according to any one of claims 1 to 7.

Images