A method and device for acquiring target domain audio training data

Abstract

The present application discloses a method and device for acquiring target domain audio training data, the method comprising: acquiring a trained source domain speech recognition model, and target domain audio data without sound event labels, the length of the audio data processed by the source domain speech recognition model being the length of the source domain data, editing the target domain audio data to the length of the source domain data, inputting the edited audio data into the source domain speech recognition model, obtaining the sound event labels of the edited audio data output by the model, editing the audio data determined to have sound event labels to the data length of the target domain audio data, and obtaining the target domain audio training data. It can be seen that the target domain audio data is length edited, the edited target domain audio data is labeled by the source domain model, and finally restored to the length of the target domain audio data, and the labeled target domain audio training data is obtained, which obtains more target domain audio training data and is more accurate.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and more specifically, to a method and device for acquiring target domain audio training data. Background Art

[0002] With the continuous development of speech recognition technology, speech recognition models have been applied in multiple scenarios, and can identify the type of speech based on the content of the speech. For example, in the live broadcast review scenario, the speech recognition model can identify malicious insults and remind the network administrator to ban them; in the smart home scenario, the speech recognition model can identify the crying of babies and remotely remind parents; in the zoo scenario, the speech recognition model can identify the sounds of wild animals and remind tourists to be careful of wild animal attacks.

[0003] Generally, speech recognition models in different scenarios will have different data length requirements for the recognized audio data. Therefore, the audio data of the existing scenario (source domain) may not meet the training data requirements of the speech recognition model of the target scenario (target domain), and the training data of the speech recognition model in the target domain is inaccurate. At the same time, the unlabeled audio data in the target domain requires a lot of manpower and time to label, so there is a lack of training data for the speech recognition model in the target domain.

[0004] How to obtain more and more accurate training data for the target domain speech recognition model is a problem that needs to be solved. Summary of the invention

[0005] In view of the above problems, the present application is proposed to provide a method and device for acquiring target domain audio training data, so as to obtain more and more accurate training data for the target domain speech recognition model.

[0006] In order to achieve the above objectives, the specific plan is proposed as follows:

[0007] A method for acquiring target domain audio training data, comprising:

[0008] Obtaining a trained source domain speech recognition model and target domain audio data without sound event labels, wherein the length of the audio data processed by the source domain speech recognition model is the length of the source domain data;

[0009] Editing the target domain audio data to the length of the source domain data to obtain edited audio data;

[0010] Inputting the edited audio data into the source domain speech recognition model to obtain a sound event label of the edited audio data output by the model;

[0011] The audio data with the sound event label is edited to the data length of the target domain audio data to obtain the target domain audio training data.

[0012] Optionally, editing the target domain audio data to the length of the source domain data to obtain edited audio data includes:

[0013] If the data length of the target domain audio data is not greater than the source domain data length, fill the target domain audio data with blank audio to the length of the source domain data to obtain target domain audio data with the data length of the source domain data;

[0014] If the data length of the target domain audio data is greater than the data length of the source domain data, the target domain audio data is divided into a plurality of sub-audio data with a data length equal to the length of the source domain data to obtain target domain audio data with a data length equal to the length of the source domain data.

[0015] Optionally, the step of editing the audio data having the sound event label to the data length of the target domain audio data to obtain the target domain audio training data includes:

[0016] If the data length of the target domain audio data is not greater than the source domain data length, delete blank audio from the audio data determined to have a sound event label to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data;

[0017] If the data length of the target domain audio data is greater than the length of the source domain data, the audio data with sound event labels will be determined, and blank audio will be filled to the data length of the target domain audio data to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data.

[0018] Optionally, the method further includes:

[0019] Acquire source domain audio data with a sound event tag, wherein the data length of the source domain audio data is the source domain data length;

[0020] If the data length of the source domain audio data is not greater than the data length of the target domain audio data, the source domain audio data is filled with blank audio to the data length of the target domain audio data to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data.

[0021] Optionally, the method further includes:

[0022] If the data length of the source domain audio data is greater than the data length of the target domain audio data, the source domain data is divided into a plurality of sub-audio data pieces each having a data length equal to the data length of the target domain audio data;

[0023] Filling the plurality of sub-audio data with blank audio to the length of the source domain data, and inputting the filled audio data into the source domain speech recognition model to obtain a sound event label of the audio data after the blank audio is filled output by the model;

[0024] For the audio data determined to have sound event labels, blank audio is deleted to obtain audio data with a data length equal to the data length of the target domain audio data, and the audio data is used as the target domain audio training data.

[0025] Optionally, after acquiring the target domain audio data without the sound event label, the method further includes:

[0026] Using the target domain audio data to train a target domain speech recognition backbone network;

[0027] The target domain audio training data is used to train a target domain speech recognition model including the target domain speech recognition backbone network.

[0028] Optionally, using the target domain audio data to train a target domain speech recognition backbone network includes:

[0029] Converting the target domain audio data into a target domain speech spectrogram, inputting the target domain speech spectrogram into an existing speech spectrum encoder, and obtaining encoded data output by the speech spectrum encoder;

[0030] Decoding the encoded data to obtain a reconstructed speech spectrogram, wherein the number of rows and columns of the reconstructed speech spectrogram is the same as the number of rows and columns of the target domain speech spectrogram;

[0031] Calculating a mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram;

[0032] The mean square error is fed back to the speech spectrum encoder so that the speech spectrum encoder can adjust the encoding mode until the mean square error meets the set convergence condition, and the finally adjusted speech spectrum encoder is used as the target domain speech recognition backbone network.

[0033] Optionally, calculating a mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram includes:

[0034] The mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram is obtained using the following loss function formula:

[0035]

[0036] Wherein, m is the number of rows of the target domain speech spectrogram, n is the number of columns of the target domain speech spectrogram, i is the row coordinate of the target domain speech spectrogram, j is the column coordinate of the target domain speech spectrogram, and y is the column coordinate of the target domain speech spectrogram. i,j is the pixel value of the i-th row and j-th column of the target domain speech spectrogram, is the pixel value of the i-th row and j-th column of the reconstructed speech spectrogram.

[0037] A device for acquiring target domain audio training data, comprising:

[0038] A target domain audio data acquisition unit, used to acquire a trained source domain speech recognition model and target domain audio data without sound event labels, wherein the length of the audio data processed by the source domain speech recognition model is the length of the source domain data;

[0039] A data length editing unit, used for editing the target domain audio data to the source domain data length to obtain edited audio data;

[0040] a label determination unit, configured to input the edited audio data into the source domain speech recognition model to obtain a sound event label of the edited audio data output by the model;

[0041] The audio training data acquisition unit is used to edit the audio data with sound event labels to the data length of the target domain audio data to obtain the target domain audio training data.

[0042] Optionally, the data length editing unit includes:

[0043] A first data filling unit is used to fill the target domain audio data with blank audio to the length of the source domain data if the data length of the target domain audio data is not greater than the source domain data length, so as to obtain target domain audio data with the data length of the source domain data;

[0044] The first data segmentation unit is used to segment the target domain audio data into a plurality of sub-audio data with a data length equal to the length of the source domain data if the data length of the target domain audio data is greater than the source domain data length, so as to obtain target domain audio data with a data length equal to the length of the source domain data.

[0045] Optionally, the audio training data acquisition unit includes:

[0046] A first data deleting unit is configured to, if the data length of the target domain audio data is not greater than the source domain data length, delete blank audio from the audio data determined to have a sound event label, and obtain audio data having a data length equal to the data length of the target domain audio data as target domain audio training data;

[0047] The second data filling unit is used to fill the audio data with sound event labels with blank audio to the data length of the target domain audio data if the data length of the target domain audio data is greater than the source domain data length, so as to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data.

[0048] Optionally, the device further comprises:

[0049] A source domain data acquisition unit, used to acquire source domain audio data with a sound event tag, wherein the data length of the source domain audio data is the source domain data length;

[0050] The third data filling unit is used to fill the source domain audio data with blank audio to the data length of the target domain audio data if the data length of the source domain audio data is not greater than the data length of the target domain audio data, so as to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data.

[0051] Optionally, the device further comprises:

[0052] A second data segmentation unit is configured to segment the source domain data into a plurality of sub-audio data pieces each having a data length equal to the data length of the target domain audio data if the data length of the source domain audio data is greater than the data length of the target domain audio data;

[0053] A fourth data filling unit is used to fill the plurality of sub-audio data with blank audio to the length of the source domain data, and input the filled audio data into the source domain speech recognition model to obtain a sound event label of the audio data after the blank audio is filled output by the model;

[0054] The second data deletion unit is used to delete blank audio from the audio data determined to have sound event labels, obtain audio data with a data length equal to the data length of the target domain audio data, and use it as target domain audio training data.

[0055] Optionally, the device further comprises:

[0056] A backbone network training unit, configured to, after acquiring the target domain audio data without the sound event labels, train a target domain speech recognition backbone network using the target domain audio data;

[0057] A model training unit is used to train a target domain speech recognition model including the target domain speech recognition backbone network using the target domain audio training data.

[0058] Optionally, the backbone network training unit includes:

[0059] A data encoding unit, used for converting the target domain audio data into a target domain speech spectrogram, inputting the target domain speech spectrogram into an existing speech spectrum encoder, and obtaining encoded data output by the speech spectrum encoder;

[0060] A data decoding unit, used for decoding the encoded data to obtain a reconstructed speech spectrogram, wherein the number of rows and columns of the reconstructed speech spectrogram is the same as the number of rows and columns of the target domain speech spectrogram;

[0061] A mean square error calculation unit, used to calculate the mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram;

[0062] The backbone network determination unit is used to feed back the mean square error to the speech spectrum encoder so that the speech spectrum encoder can adjust the encoding method until the mean square error meets the set convergence condition, and the finally adjusted speech spectrum encoder is used as the target domain speech recognition backbone network.

[0063] Optionally, the mean square error calculation unit includes:

[0064] The mean square error calculation subunit is used to obtain the mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram using the following loss function formula:

[0065]

[0066] Wherein, m is the number of rows of the target domain speech spectrogram, n is the number of columns of the target domain speech spectrogram, i is the row coordinate of the target domain speech spectrogram, j is the column coordinate of the target domain speech spectrogram, and y is the column coordinate of the target domain speech spectrogram. i,j is the pixel value of the i-th row and j-th column of the target domain speech spectrogram, is the pixel value of the i-th row and j-th column of the reconstructed speech spectrogram.

[0067] By means of the above technical solution, the present application obtains a trained source domain speech recognition model and target domain audio data without sound event labels. The length of the audio data processed by the source domain speech recognition model is the length of the source domain data. Further, the target domain audio data is edited to the length of the source domain data to obtain the edited audio data. The edited audio data is input into the source domain speech recognition model to obtain the sound event labels of the edited audio data output by the model. The audio data with sound event labels is edited to the data length of the target domain audio data to obtain the target domain audio training data. It can be seen that the target domain audio data is length edited, the edited target domain audio data is labeled by the source domain model, and finally restored to the length of the target domain audio data. The target domain audio training data with labels is obtained, and more target domain audio training data is obtained while being more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0069] Figure 1 A schematic diagram of a process for obtaining target domain audio training data provided in an embodiment of the present application;

[0070] Figure 2 A schematic diagram of a process for obtaining target domain audio training data based on source domain data provided in an embodiment of the present application;

[0071] Figure 3 A schematic diagram of the structure of a device for obtaining target domain audio training data provided in an embodiment of the present application;

[0072] Figure 4 A schematic diagram of the structure of a device for acquiring target domain audio training data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] The present application solution can be implemented based on a terminal with data processing capabilities, which can be a computer, server, cloud, etc.

[0075] Next, combine Figure 1 The method for acquiring target domain audio training data of the present application may include the following steps:

[0076] Step S110: obtaining a trained source domain speech recognition model and target domain audio data without sound event labels, wherein the length of the audio data processed by the source domain speech recognition model is the length of the source domain data.

[0077] It can be understood that the source domain can represent a specific scene, so the source domain speech recognition model can judge the speech events more accurately in this specific scene, and the speech data in the source domain has commonality in length, so the source domain speech recognition model can be trained with labeled speech data of common length, then the common length can be the optimal data length of the source domain speech recognition model.

[0078] Specifically, the source domain speech recognition model can process audio data of optimal data length and output speech event labels of the input audio data.

[0079] Step S120: Edit the target domain audio data to the length of the source domain data to obtain edited audio data.

[0080] It is understandable that the target domain audio data input into the source domain speech recognition model needs to meet the length requirement of the source domain speech recognition model for processing audio data.

[0081] Specifically, the editing process of the target domain audio data can be to fill the target domain audio data that is shorter than the source domain data length, or to split the target domain audio data that is longer than the source domain data length, so that the edited audio data meets the length requirement of the source domain speech recognition model for processing audio data.

[0082] Step S130: input the edited audio data into the source domain speech recognition model to obtain the sound event label of the edited audio data output by the model.

[0083] Specifically, after processing the target domain audio data, the audio data that meets the length requirement can be input into the source domain speech recognition model, and the confidence of each sound event that can be recognized by the model can be determined, and it can be judged whether the confidence meets the preset first threshold range to determine whether to determine the label of the sound event.

[0084] The preset first threshold range may represent a range of reasonable confidence values ​​for the sound event.

[0085] For example, if audio data X that meets the length requirement is input into the model, and the model outputs a confidence of 3% for "contains sound A", then the confidence of "does not contain sound A" is 97%, and the preset threshold range of the "does not contain sound A" event can be 0-5%, and the preset threshold range of the "contains sound A" event can be 95%-100%, then the label "does not contain sound A" can be annotated to the audio data X. For another example, if the model outputs a confidence of 20% for "contains sound A", then this confidence is neither within the preset threshold range of the "does not contain sound A" event nor within the preset threshold range of the "contains sound A" event, then the audio data X may not be annotated with a label about sound event A, and the above method of determining the sound event label may be repeated based on the confidence of the next sound event (sound event B).

[0086] Step S140: Edit the audio data that is determined to have the sound event label to the data length of the target domain audio data to obtain the target domain audio training data.

[0087] Specifically, after determining the label of each sound event, the edited audio data can be labeled with the label of each sound event to obtain audio data with determined sound event labels. For the edited and labeled audio data, if its data length is longer than the data length of the target domain audio data, it can be trimmed; if its data length is shorter than the data length of the target domain audio data, it can be padded until its length meets the data length of the target domain audio data.

[0088] The method for acquiring target domain audio training data provided in this embodiment can edit the length of the target domain audio data, label the edited target domain audio data through the source domain model, and finally restore it to the length of the target domain audio data. The obtained labeled target domain audio training data is used to train the target domain speech recognition model, thereby obtaining more target domain audio training data and being more accurate.

[0089] It can be understood that the target domain can refer to any scene outside the source domain, so the optimal audio data length of the target domain may be longer or shorter than the source domain data length. Based on this, in some embodiments of the present application, the above step S130, editing the target domain audio data to the source domain data length to obtain the edited audio data, is introduced. The process may include the following two cases:

[0090] The first method is to fill the target domain audio data with blank audio to the length of the source domain data if the data length of the target domain audio data is not greater than the length of the source domain data, so as to obtain target domain audio data having a data length equal to the length of the source domain data.

[0091] It is understandable that when the data length of the target domain audio data is not greater than the source domain data length, it is necessary to supplement the data length of the target domain audio data to the source domain data length. The specific process of supplementation can be to fill the target domain audio data with blank audio, and the filling can be performed at any position of the target domain audio data, so that the length of each audio data filled is the length of the source domain data.

[0092] For example, the length of source domain data may be 10s, and the length of target domain audio data may be 7s. 3s of blank audio may be padded before the 0th second of the target domain audio data, or 3s of blank audio may be padded after the 7th second of the target domain audio data to obtain 10s of audio data of 7s of valid audio + 3s of blank audio.

[0093] The second type is that if the data length of the target domain audio data is greater than the length of the source domain data, the target domain audio data is divided into a plurality of sub-audio data with a data length equal to the length of the source domain data to obtain target domain audio data with a data length equal to the length of the source domain data.

[0094] It is understandable that when the data length of the target domain audio data is greater than the source domain data length, it is necessary to reduce the data length of the target domain audio data to the source domain data length. The specific process of reduction can be to cut the target domain audio data into several segments, each segment of audio data has a length of the source domain data, wherein every two segments of audio data can contain repeated audio parts. The specific process of reduction can also be to cut the target domain audio data into several segments, each segment of audio data is not repeated, but each segment of audio data is not longer than the source domain data length, then fill each segment of audio data with blank audio to the source domain data length.

[0095] For example, the source domain data length may be 10s, and the target domain audio data length may be 14s. The target domain audio data may be trimmed into two segments: the first segment (0s to 10s) and the second segment (4s to 14s), or the first segment (0s to 7s) and the second segment (7s to 14s), and then the audio data of 0s to 7s and 7s to 14s may be padded with 3s of blank audio to obtain 10s of audio data of 7s of valid audio + 3s of blank audio.

[0096] The method for acquiring target domain audio training data provided in this embodiment can process target domain audio data that is longer or shorter than the length of source domain data, and obtain audio data that meets the data length requirements for processing by the source domain speech recognition model, so as to input the source domain speech recognition model.

[0097] It can be understood that after editing the target domain audio data that is longer / shorter than the source domain data length, it is necessary to restore it to the data length of the target domain audio data. Based on this, in some embodiments of the present application, the above step S140, editing the audio data determined to have sound event labels to the data length of the target domain audio data to obtain the target domain audio training data is introduced. This process may include the following two cases:

[0098] 1) If the data length of the target domain audio data is not greater than the length of the source domain data, delete the blank audio from the audio data determined to have a sound event label to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data.

[0099] Specifically, when the data length of the target domain audio data is not greater than the length of the source domain data, the audio data input to the source domain speech recognition model can be the audio data after the target domain audio data is filled with blank audio. Then, after the model outputs the sound event label and marks it, the filled blank audio can be deleted to obtain the target domain audio training data.

[0100] For example, the data length of the target domain audio data can be 7s, and the audio data is edited to 10s of 7s of valid audio + 3s of blank audio before input. Then, after the model outputs the sound event label and annotates it, the 3s of blank audio can be deleted from the labeled audio to obtain 7s of labeled audio data as the target domain audio training data.

[0101] 2) If the data length of the target domain audio data is greater than the length of the source domain data, the audio data with sound event labels will be determined, and blank audio will be filled to the data length of the target domain audio data to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data.

[0102] Specifically, when the data length of the target domain audio data is not greater than the length of the source domain data, the audio data input into the source domain speech recognition model can be the audio data after the target domain audio data is trimmed. Then, after the model outputs the sound event label and marks it, the blank audio can be filled to the length of the target domain audio data to obtain the target domain audio training data.

[0103] For example, the data length of the target domain audio data can be 14s, and the 10s audio data is edited into two segments of 7s valid audio + 3s blank audio before input. Then, after the model outputs the sound event label and annotates it, for each segment of labeled audio, 4s of blank audio can be supplemented to 14s, and 14s of labeled audio data can be obtained as the target domain audio training data.

[0104] The method for acquiring target domain audio training data provided in this embodiment can process the labeled audio data output by the model to obtain audio data that meets the length of the target domain audio data to train the target domain speech recognition model.

[0105] In some embodiments of the present application, considering adding more data to train the target domain speech recognition model, the process of processing the source domain audio data to obtain the target domain audio training data can be further increased, specifically as follows: Figure 2 As shown, the process may include:

[0106] Step S201: Acquire source domain audio data with sound event tags.

[0107] Specifically, the source domain audio data may represent existing audio data with event tags in a source domain scenario, wherein the data length of the source domain audio data is the source domain data length.

[0108] Step S202, determine whether the data length of the source domain audio data is greater than the data length of the target domain audio data, if not, execute step S203, if yes, execute step S204.

[0109] It can be understood that the length of the final output audio data needs to meet the data length requirement of the target domain audio data, and the data length editing method of the source domain audio data that is longer or shorter than the data length of the target domain audio data is different. Therefore, when the data length of the source domain audio data is greater than the data length of the target domain audio data, step S204 is executed, and when the data length of the source domain audio data is not greater than the data length of the target domain audio data, step S203 is executed.

[0110] Step S203: fill the source domain audio data with blank audio to the data length of the target domain audio data, and obtain audio data with a data length equal to the data length of the target domain audio data as target domain audio training data.

[0111] It is understandable that the label of the source domain audio data can indicate the sound event of the source domain audio data, and no matter how much blank audio is filled in, it represents the original sound event, that is, the label remains unchanged. Therefore, when the data length of the source domain audio data is not greater than the data length of the target domain audio data, the source domain audio data can be directly filled with blank audio to the data length of the target audio data, and the original label is retained to obtain the target domain audio training data.

[0112] Step S204: divide the source domain data into a plurality of sub-audio data pieces whose data length is the same as the data length of the target domain audio data.

[0113] Specifically, the segmentation process may be to segment the source domain audio data into a plurality of segments, the length of each segment of audio data may be the data length of the target domain audio data, wherein every two segments of audio data may contain repeated audio parts. The segmentation process may also be to segment the source domain audio data into a plurality of segments, each segment of audio data is not repeated, but each segment of audio data is not longer than the data length of the target domain audio data.

[0114] For example, the length of source domain data may be 14s, and the length of target domain audio data may be 10s. The source domain audio data may be trimmed into two segments: the first segment (0s to 10s) and the second segment (4s to 14s), or the first segment (0s to 7s) and the second segment (7s to 14s).

[0115] Step S205, fill the plurality of sub-audio data with blank audio to the length of the source domain data, and input the filled audio data into the source domain speech recognition model to obtain the sound event label of the audio data after the blank audio is filled output by the model.

[0116] Specifically, after processing the source domain audio data, the audio data that meets the length requirement can be input into the source domain speech recognition model, the confidence of each sound event that the model can recognize can be determined, and it is judged whether the confidence meets the preset second threshold range to determine whether to determine the label of the sound event.

[0117] It can be understood that the audio data in the target domain is unlabeled audio data. In order to ensure the high accuracy of labeling the target domain audio data, the first threshold range needs to be more stringent than the second threshold range. That is, if the second threshold range for "containing a certain sound event" is 80% to 100%, then the first threshold range can be narrower: 95% to 100%. Correspondingly, if the second threshold range for "not containing a certain sound event" is 0 to 20%, then the first threshold range can be narrower: 0 to 50%.

[0118] For example, the length of source domain data may be 14s, and the length of target domain audio data may be 10s. The source domain audio data may be trimmed into a first segment (0th to 7th s) and a second segment (7th to 14th s), and then the audio data of 0th to 7th s and 7th to 14th s may be filled with 7s of blank audio to obtain 14s of audio data of 7s of valid audio + 7s of blank audio, and one of the 14s audio segments may be recorded as audio data Y.

[0119] The audio data Y that meets the length requirement is input into the model, and the confidence of the model output "contains sound A" is 10%, then the confidence of "does not contain sound A" is 90%, and the preset threshold range of the "does not contain sound A" event can be 0-20%, and the preset threshold range of the "contains sound A" event can be 80%-100%, then the label "does not contain sound A" can be marked on the audio data X. For another example, if the confidence of the model output "contains sound A" is 30%, then this confidence is neither within the preset threshold range of the "does not contain sound A" event nor within the preset threshold range of the "contains sound A" event, then the label of sound event A can be not marked on the audio data X, and the above method of determining the sound event label can be repeated according to the confidence of the next sound event (sound event B).

[0120] Step S206: Delete blank audio from the audio data that is determined to have a sound event label, obtain audio data with a data length equal to the data length of the target domain audio data, and use it as target domain audio training data.

[0121] Specifically, the length of the deleted blank audio may be the difference between the length of the source domain audio data and the length of the target domain audio data.

[0122] For example, the length of source domain data can be 14s, and the data length of target domain audio data can be 10s. For the 14s audio data with sound event labels of 7s valid audio + 7s blank audio, 4s blank audio can be deleted to obtain 10s audio data with sound event labels of 7s valid audio + 3s blank audio as the target domain audio training data.

[0123] The method for acquiring target domain audio training data provided in this embodiment can analyze the relationship between the source domain audio length and the data length of the target domain audio data, process the source domain audio data, obtain the target domain audio training data, and enrich the training data of the target domain speech recognition model.

[0124] The foregoing embodiments describe how to obtain target domain audio training data. In some embodiments of the present application, considering the use of existing and processed audio data to train a target domain speech recognition model, an embodiment of the present application provides a method for training a target domain speech recognition model. After obtaining target domain audio data without sound event labels mentioned in the above embodiments, a process for training the target domain speech recognition model may be added. Specifically, the process may include:

[0125] S1. Utilize the target domain audio data to train a target domain speech recognition backbone network.

[0126] Specifically, a self-supervised learning method can be used to train the backbone network of the target domain speech recognition model using unlabeled target domain audio data, which can specifically include the following steps:

[0127] S11. Convert the target domain audio data into a target domain speech spectrogram, input the target domain speech spectrogram into an existing speech spectrum encoder, and obtain encoded data output by the speech spectrum encoder.

[0128] Specifically, the target domain speech spectrogram can represent the components of various sound events, so that the speech spectrum encoder can extract the features of the target domain sound events therefrom.

[0129] S12. Decode the encoded data to obtain a reconstructed speech spectrogram.

[0130] The number of rows and columns of the reconstructed speech spectrogram is the same as that of the target domain speech spectrogram.

[0131] S13. Calculate the mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram.

[0132] Specifically, the size of the mean square error can represent the effect of the speech spectrum encoder in extracting the features of the target domain sound events. The smaller the mean square error, the better the effect of the speech spectrum encoder in extracting the features of the target domain sound events.

[0133] The following loss function formula can be used to obtain the mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram:

[0134]

[0135] Wherein, m is the number of rows of the target domain speech spectrogram, n is the number of columns of the target domain speech spectrogram, i is the row coordinate of the target domain speech spectrogram, j is the column coordinate of the target domain speech spectrogram, and y is the column coordinate of the target domain speech spectrogram. i,j is the pixel value of the i-th row and j-th column of the target domain speech spectrogram, is the pixel value of the i-th row and j-th column of the reconstructed speech spectrogram.

[0136] S14, feeding back the mean square error to the speech spectrum encoder so that the speech spectrum encoder can adjust the encoding mode until the mean square error meets the set convergence condition, and using the finally adjusted speech spectrum encoder as the target domain speech recognition backbone network.

[0137] Specifically, the convergence condition may represent a maximum value of a mean square error, and the speech spectrum encoder after final adjustment may be able to recognize key features of the target domain audio with high accuracy.

[0138] S2. Using the target domain audio training data, train a target domain speech recognition model including the target domain speech recognition backbone network.

[0139] Specifically, the target domain speech recognition model may include a backbone network and a classification network. After the backbone network capable of extracting key features of the target domain audio is trained, the classification network constituting the target domain speech recognition model needs to be trained. The target domain audio training data obtained by processing may be used to train the target domain speech recognition model. The target domain audio data without sound event labels may be input into the trained target domain speech recognition model, and each sound event and the confidence corresponding to each sound event may be output.

[0140] The method for acquiring target domain audio training data provided in this embodiment can use unlabeled target domain audio data to train a target domain speech recognition backbone network, and on this basis, use the target domain audio training data to train a target domain speech recognition model that includes the target domain speech recognition backbone network. It can be seen that the backbone network for training the target domain speech recognition model uses a large amount of unlabeled audio data, so that the backbone network has a higher accuracy rate in identifying key features of the target domain audio. The target domain speech recognition model uses the target domain audio training data obtained based on the source domain audio data and the target domain audio data. The audio data used for training is richer, making the trained model more accurate.

[0141] The following is a description of an apparatus for acquiring target domain audio training data provided in an embodiment of the present application. The apparatus for acquiring target domain audio training data described below and the method for acquiring target domain audio training data described above can be referenced to each other.

[0142] See also Figure 3 , Figure 3 A schematic diagram of the structure of a device for acquiring target domain audio training data disclosed in an embodiment of the present application.

[0143] like Figure 3 As shown, the device may include:

[0144] A target domain audio data acquisition unit 11 is used to acquire a trained source domain speech recognition model and target domain audio data without sound event labels, wherein the length of the audio data processed by the source domain speech recognition model is the length of the source domain data;

[0145] A data length editing unit 12, used for editing the target domain audio data to the source domain data length to obtain edited audio data;

[0146] A label determination unit 13, configured to input the edited audio data into the source domain speech recognition model to obtain a sound event label of the edited audio data output by the model;

[0147] The audio training data acquisition unit 14 is used to edit the audio data with sound event tags to the data length of the target domain audio data to obtain the target domain audio training data.

[0148] Optionally, the data length editing unit 12 includes:

[0149] A first data filling unit is used to fill the target domain audio data with blank audio to the length of the source domain data if the data length of the target domain audio data is not greater than the source domain data length, so as to obtain target domain audio data with the data length of the source domain data;

[0150] The first data segmentation unit is used to segment the target domain audio data into a plurality of sub-audio data with a data length equal to the length of the source domain data if the data length of the target domain audio data is greater than the source domain data length, so as to obtain target domain audio data with a data length equal to the length of the source domain data.

[0151] Optionally, the audio training data acquiring unit 14 includes:

[0152] A first data deleting unit is configured to, if the data length of the target domain audio data is not greater than the source domain data length, delete blank audio from the audio data determined to have a sound event label, and obtain audio data having a data length equal to the data length of the target domain audio data as target domain audio training data;

[0153] The second data filling unit is used to fill the audio data with sound event labels with blank audio to the data length of the target domain audio data if the data length of the target domain audio data is greater than the source domain data length, so as to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data.

[0154] Optionally, the device further comprises:

[0155] A source domain data acquisition unit, used to acquire source domain audio data with a sound event tag, wherein the data length of the source domain audio data is the source domain data length;

[0156] The third data filling unit is used to fill the source domain audio data with blank audio to the data length of the target domain audio data if the data length of the source domain audio data is not greater than the data length of the target domain audio data, so as to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data.

[0157] Optionally, the device further comprises:

[0158] A second data segmentation unit is configured to segment the source domain data into a plurality of sub-audio data pieces each having a data length equal to the data length of the target domain audio data if the data length of the source domain audio data is greater than the data length of the target domain audio data;

[0159] A fourth data filling unit is used to fill the plurality of sub-audio data with blank audio to the length of the source domain data, and input the filled audio data into the source domain speech recognition model to obtain a sound event label of the audio data after the blank audio is filled output by the model;

[0160] The second data deletion unit is used to delete blank audio from the audio data determined to have sound event labels, obtain audio data with a data length equal to the data length of the target domain audio data, and use it as target domain audio training data.

[0161] Optionally, the device further comprises:

[0162] A backbone network training unit, configured to, after acquiring the target domain audio data without the sound event labels, train a target domain speech recognition backbone network using the target domain audio data;

[0163] A model training unit is used to train a target domain speech recognition model including the target domain speech recognition backbone network using the target domain audio training data.

[0164] Optionally, the backbone network training unit includes:

[0165] A data encoding unit, used for converting the target domain audio data into a target domain speech spectrogram, inputting the target domain speech spectrogram into an existing speech spectrum encoder, and obtaining encoded data output by the speech spectrum encoder;

[0166] A data decoding unit, used for decoding the encoded data to obtain a reconstructed speech spectrogram, wherein the number of rows and columns of the reconstructed speech spectrogram is the same as the number of rows and columns of the target domain speech spectrogram;

[0167] A mean square error calculation unit, used to calculate the mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram;

[0168] The backbone network determination unit is used to feed back the mean square error to the speech spectrum encoder so that the speech spectrum encoder can adjust the encoding method until the mean square error meets the set convergence condition, and the finally adjusted speech spectrum encoder is used as the target domain speech recognition backbone network.

[0169] Optionally, the mean square error calculation unit includes:

[0170] The mean square error calculation subunit is used to obtain the mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram using the following loss function formula:

[0171]

[0172] Wherein, m is the number of rows of the target domain speech spectrogram, n is the number of columns of the target domain speech spectrogram, i is the row coordinate of the target domain speech spectrogram, j is the column coordinate of the target domain speech spectrogram, and y is the column coordinate of the target domain speech spectrogram. i,j is the pixel value of the i-th row and j-th column of the target domain speech spectrogram, is the pixel value of the i-th row and j-th column of the reconstructed speech spectrogram.

[0173] The device for acquiring target domain audio training data provided in the embodiment of the present application can be applied to a device for acquiring target domain audio training data, such as a terminal: a mobile phone, a computer, etc. Optionally, Figure 4 The hardware structure diagram of the acquisition of target domain audio training data is shown in FIG. Figure 4 , the hardware structure of the target domain audio training data acquisition device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0174] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0175] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0176] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0177] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to:

[0178] Obtaining a trained source domain speech recognition model and target domain audio data without sound event labels, wherein the length of the audio data processed by the source domain speech recognition model is the length of the source domain data;

[0179] Editing the target domain audio data to the length of the source domain data to obtain edited audio data;

[0180] Inputting the edited audio data into the source domain speech recognition model to obtain a sound event label of the edited audio data output by the model;

[0181] The audio data with the sound event label is edited to the data length of the target domain audio data to obtain the target domain audio training data.

[0182] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0183] The embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0184] Obtaining a trained source domain speech recognition model and target domain audio data without sound event labels, wherein the length of the audio data processed by the source domain speech recognition model is the length of the source domain data;

[0185] Editing the target domain audio data to the length of the source domain data to obtain edited audio data;

[0186] Inputting the edited audio data into the source domain speech recognition model to obtain a sound event label of the edited audio data output by the model;

[0187] The audio data with the sound event label is edited to the data length of the target domain audio data to obtain the target domain audio training data.

[0188] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0189] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0190] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0191] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for acquiring target domain audio training data, characterized in that: include: Obtaining a trained source domain speech recognition model and target domain audio data without sound event labels, wherein the length of the audio data processed by the source domain speech recognition model is the length of the source domain data; Editing the target domain audio data to the length of the source domain data to obtain edited audio data; Inputting the edited audio data into the source domain speech recognition model to obtain a sound event label of the edited audio data output by the model; The audio data with the sound event label is edited to the data length of the target domain audio data to obtain the target domain audio training data.

2. The method according to claim 1, characterized in that Editing the target domain audio data to the length of the source domain data to obtain edited audio data includes: If the data length of the target domain audio data is not greater than the source domain data length, fill the target domain audio data with blank audio to the length of the source domain data to obtain target domain audio data with the data length of the source domain data; If the data length of the target domain audio data is greater than the data length of the source domain data, the target domain audio data is divided into a plurality of sub-audio data with a data length equal to the length of the source domain data to obtain target domain audio data with a data length equal to the length of the source domain data.

3. The method according to claim 2, characterized in that The step of editing the audio data having the sound event label to the data length of the target domain audio data to obtain the target domain audio training data comprises: If the data length of the target domain audio data is not greater than the source domain data length, delete blank audio from the audio data determined to have a sound event label to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data; If the data length of the target domain audio data is greater than the length of the source domain data, the audio data with sound event labels will be determined, and blank audio will be filled to the data length of the target domain audio data to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data.

4. The method according to claim 1, characterized in that: Also includes: Acquire source domain audio data with a sound event tag, wherein the data length of the source domain audio data is the source domain data length; If the data length of the source domain audio data is not greater than the data length of the target domain audio data, the source domain audio data is filled with blank audio to the data length of the target domain audio data to obtain audio data with a data length equal to the data length of the target domain audio data as the target domain audio training data.

5. The method according to claim 4, characterized in that Also includes: If the data length of the source domain audio data is greater than the data length of the target domain audio data, the source domain data is divided into a plurality of sub-audio data pieces each having a data length equal to the data length of the target domain audio data; Filling the plurality of sub-audio data with blank audio to the length of the source domain data, and inputting the filled audio data into the source domain speech recognition model to obtain a sound event label of the audio data after the blank audio is filled output by the model; For the audio data determined to have sound event labels, blank audio is deleted to obtain audio data with a data length equal to the data length of the target domain audio data, and the audio data is used as the target domain audio training data.

6. The method according to any one of claims 1 to 5, characterized in that: After obtaining the target domain audio data without sound event labels, it also includes: Using the target domain audio data to train a target domain speech recognition backbone network; The target domain audio training data is used to train a target domain speech recognition model including the target domain speech recognition backbone network.

7. The method according to claim 6, characterized in that The target domain audio data is used to train a target domain speech recognition backbone network, including: Converting the target domain audio data into a target domain speech spectrogram, inputting the target domain speech spectrogram into an existing speech spectrum encoder, and obtaining encoded data output by the speech spectrum encoder; Decoding the encoded data to obtain a reconstructed speech spectrogram, wherein the number of rows and columns of the reconstructed speech spectrogram is the same as the number of rows and columns of the target domain speech spectrogram; Calculating a mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram; The mean square error is fed back to the speech spectrum encoder so that the speech spectrum encoder can adjust the encoding mode until the mean square error meets the set convergence condition, and the finally adjusted speech spectrum encoder is used as the target domain speech recognition backbone network.

8. The method according to claim 7, characterized in that Calculating a mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram, comprising: The mean square error between the target domain speech spectrogram and the reconstructed speech spectrogram is obtained using the following loss function formula: Wherein, m is the number of rows of the target domain speech spectrogram, n is the number of columns of the target domain speech spectrogram, i is the row coordinate of the target domain speech spectrogram, j is the column coordinate of the target domain speech spectrogram, and y is the column coordinate of the target domain speech spectrogram. i,j is the pixel value of the i-th row and j-th column of the target domain speech spectrogram, is the pixel value of the i-th row and j-th column of the reconstructed speech spectrogram.

9. A device for acquiring target domain audio training data, characterized in that: include: A target domain audio data acquisition unit, used to acquire a trained source domain speech recognition model and target domain audio data without sound event labels, wherein the length of the audio data processed by the source domain speech recognition model is the length of the source domain data; A data length editing unit, used for editing the target domain audio data to the source domain data length to obtain edited audio data; a label determination unit, configured to input the edited audio data into the source domain speech recognition model to obtain a sound event label of the edited audio data output by the model; The audio training data acquisition unit is used to edit the audio data with sound event labels to the data length of the target domain audio data to obtain the target domain audio training data.

10. The device according to claim 9, characterized in that The data length editing unit comprises: A data filling unit, configured to fill the target domain audio data with blank audio to the length of the source domain data if the data length of the target domain audio data is not greater than the source domain data length, so as to obtain target domain audio data with the same data length as the source domain data length; A data segmentation unit is used to segment the target domain audio data into a plurality of sub-audio data with a data length equal to the length of the source domain data if the data length of the target domain audio data is greater than the source domain data length, so as to obtain target domain audio data with a data length equal to the length of the source domain data.

Images