Model training method, audio data processing method, corresponding device and product

Abstract

The present disclosure provides a model training method, an audio data processing method, corresponding devices and products. The model training method comprises: for any round, inputting first audio training data corresponding to a preset application field into a first target labeling model and a first audio recognition model of the current round to obtain first predicted labeling data corresponding to the first audio training data; deleting target audio training data in the first audio training data according to the first predicted labeling data to obtain second audio training data; performing noise training on the first audio recognition model based on the second audio training data to obtain a second audio recognition model; training the first target labeling model based on the second audio training data and the second audio recognition model to obtain a second target labeling model of the current round; and performing model training of the next round when a preset convergence condition is not met. The embodiments of the present disclosure can make the target labeling model quickly applicable to the preset application field through training.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model training method, an audio data processing method, a model training device, an audio data processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Speech annotation, as a crucial foundational technology in artificial intelligence, plays a vital role in intelligent speech processing and is currently widely applied in scenarios such as intelligent assistants, meeting recording, customer service quality inspection, and medical transcription. However, considering the significant differences in audio data across various application scenarios, general-purpose audio annotation models often exhibit poor accuracy when processing audio data from specific application domains. Summary of the Invention

[0003] This disclosure provides a model training method, an audio data processing method, a model training device, an audio data processing device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] Firstly, this disclosure provides a model training method, which includes: for any given round, inputting first audio training data corresponding to a preset application domain into a first target labeling model and a first audio recognition model for the current round to obtain first predicted labeling data corresponding to the first audio training data, wherein the first audio recognition model is used to acquire prior information corresponding to the first audio training data, and the number of the first audio recognition models is one or more; deleting target audio training data from the first audio training data according to the first predicted labeling data to obtain second audio training data, wherein the confidence level of the first predicted labeling data corresponding to the target audio training data is less than the confidence level of the first predicted labeling data corresponding to the second audio training data; performing noise-added training on the first audio recognition model based on the second audio training data to obtain a second audio recognition model; training the first target labeling model based on the second audio training data and the second audio recognition model to obtain a second target labeling model for the current round; and, if it is determined that a preset convergence condition is not met, executing model training for the next round, wherein the second target labeling model and the second audio recognition model for the current round are respectively used as the first target labeling model and the first audio recognition model for the next round.

[0005] Secondly, this disclosure provides an audio data processing method, which includes: inputting audio data to be labeled into a target labeling model and at least one preset audio recognition model to obtain target predicted labeling data corresponding to the audio data to be labeled; wherein, the target labeling model is obtained using the model training method described in any one of the embodiments of this disclosure, the audio data to be labeled belongs to a preset application domain, the audio recognition model is used to obtain prior information corresponding to the audio data to be labeled, and inputs the prior information into the target labeling model, so that the target labeling model obtains the target predicted labeling data based on the audio data to be labeled and the prior information.

[0006] Thirdly, this disclosure provides a model training apparatus, comprising: a prediction module, configured to, for any given round, input first audio training data corresponding to a preset application domain into a first target labeling model and a first audio recognition model for the current round, to obtain first predicted labeling data corresponding to the first audio training data, wherein the first audio recognition model is used to acquire prior information corresponding to the first audio training data, and the number of the first audio recognition models is one or more; and a deletion module, configured to delete target audio training data from the first audio training data according to the first predicted labeling data, to obtain second audio training data, wherein the first predicted labeling data corresponds to the target audio training data. The confidence level of the data is less than the confidence level of the first predicted annotation data corresponding to the second audio training data; the first training module is used to perform noise-added training on the first audio recognition model based on the second audio training data to obtain the second audio recognition model; the second training module is used to train the first target annotation model based on the second audio training data and the second audio recognition model to obtain the second target annotation model for the current round; the loop module is used to execute the model training for the next round if it is determined that the preset convergence condition is not met, wherein the second target annotation model and the second audio recognition model for the current round are respectively used as the first target annotation model and the first audio recognition model for the next round.

[0007] Fourthly, this disclosure provides an audio data processing apparatus, comprising: a processing module, configured to input audio data to be labeled into a target labeling model and at least one preset audio recognition model respectively, to obtain target predicted labeling data corresponding to the audio data to be labeled; wherein, the target labeling model is obtained using the model training method described in any one of the embodiments of this disclosure, the audio data to be labeled belongs to a preset application domain, and the audio recognition model is configured to obtain prior information corresponding to the audio data to be labeled, and input the prior information into the target labeling model, so that the target labeling model obtains the target predicted labeling data based on the audio data to be labeled and the prior information.

[0008] Fifthly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the model training method or audio data processing method described above.

[0009] In a sixth aspect, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described model training method or audio data processing method.

[0010] In a seventh aspect, this disclosure provides a computer program product that includes computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the model training method or audio data processing method described above.

[0011] The model training method provided in this disclosure can quickly adjust the target annotation model to be suitable for audio annotation in a specific application domain through multiple rounds of iterative training. Specifically, for any round, the first audio training data corresponding to the preset application domain is first input into the first target annotation model and the first audio recognition model of the current round to obtain the first predicted annotation data. Then, the first predicted annotation data is filtered according to the confidence level of the first predicted annotation data, and the part of the first audio training data with lower annotation quality (i.e., the target audio training data) is deleted to obtain the second audio training data with higher annotation quality. The first audio recognition model is first trained with noise added using the second audio training data to obtain the second audio recognition model. Then, the first target annotation model is trained using the second audio training data and the second audio recognition model to obtain the second target annotation model with better performance, and then enters the next round of training. Therefore, if the initially obtained target annotation model (i.e. the target annotation model that has not yet undergone the first round of processing) does not perform well in the preset application domain, the network parameters of the target annotation model can be quickly adjusted in the above manner, so that the trained target annotation model can be quickly applied to the aforementioned preset application domain, thereby enabling the target annotation model to be applied to audio annotation in the preset application domain and achieving a higher processing effect.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0014] Figure 1 A flowchart of a model training method provided in this embodiment of the disclosure;

[0015] Figure 2 A schematic diagram illustrating a model training method provided in an embodiment of this disclosure;

[0016] Figure 3 A schematic diagram illustrating a model training method provided in an embodiment of this disclosure;

[0017] Figure 4 A schematic diagram illustrating a model training method provided in an embodiment of this disclosure;

[0018] Figure 5A flowchart of an audio data processing method provided in this disclosure embodiment;

[0019] Figure 6 A schematic diagram illustrating an audio data processing method provided in an embodiment of this disclosure;

[0020] Figure 7 A block diagram of a model training apparatus provided in an embodiment of this disclosure;

[0021] Figure 8 A block diagram of an audio data processing apparatus provided in an embodiment of this disclosure;

[0022] Figure 9 A block diagram of an electronic device provided in an embodiment of this disclosure;

[0023] Figure 10 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0026] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0028] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0029] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information in this technical solution comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example, appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely identifying specific individuals.

[0030] Audio data varies significantly depending on the application scenario. For example, audio data in customer service differs greatly from audio data in film and television works. Therefore, using a general audio annotation model to annotate audio data for specific application areas may result in poor accuracy and fail to meet usage requirements.

[0031] In view of the above, embodiments of this disclosure provide a model training method, an audio data processing method, a model training apparatus, an audio data processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0032] In this embodiment of the disclosure, the first audio training data of the preset application domain is used, and the processing capability of the audio recognition model is combined to iteratively train the first target annotation model, thereby quickly and efficiently obtaining an audio annotation model suitable for the preset application domain.

[0033] Firstly, the present disclosure provides a model training method.

[0034] Figure 1 A flowchart illustrating a model training method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The model training method includes:

[0035] Step S11: For any round, the first audio training data corresponding to the preset application domain is input into the first target annotation model and the first audio recognition model of the current round to obtain the first predicted annotation data corresponding to the first audio training data. The first audio recognition model is used to obtain the prior information corresponding to the first audio training data, and the number of the first audio recognition models is one or more.

[0036] Step S12: Delete the target audio training data in the first audio training data according to the first predicted annotation data to obtain the second audio training data. The confidence level of the first predicted annotation data corresponding to the audio training data is less than the confidence level of the first predicted annotation data corresponding to the second audio training data.

[0037] Step S13: Based on the second audio training data, the first audio recognition model is trained with noise to obtain the second audio recognition model;

[0038] Step S14: Train the first target annotation model based on the second audio training data and the second audio recognition model to obtain the second target annotation model for the current round;

[0039] Step S15: If the preset convergence condition is not met, perform the next round of model training, wherein the second target annotation model and the second audio recognition model in the current round are respectively used as the first target annotation model and the first audio recognition model in the next round.

[0040] In some optional embodiments, after obtaining the target annotation model through pre-training, if the accuracy of the annotation result is low when directly applying the target annotation model to perform audio annotation in a preset application domain, it indicates that the target annotation model has a poor application effect in the preset application domain. Based on this, the model training method of this disclosure can be adopted, using the first audio training data corresponding to the preset application domain, and combining it with the audio recognition model to iteratively train the target annotation model (i.e., the first target annotation model in the first round), thereby obtaining an audio annotation model with a better application effect in the preset application domain (i.e., the second target annotation model in the last round). The preset application domains include customer service application scenarios, film and television work application scenarios, remote consultation application scenarios, online course application scenarios, intelligent driving application scenarios, etc., and this disclosure does not limit these.

[0041] The audio recognition model includes some small-scale audio processing models that can obtain certain information related to the input audio modal data and use this information as prior information for the first target labeling model or the second target labeling model to assist the first target labeling model or the second target labeling model in performing the forward inference process.

[0042] As mentioned earlier, to improve the overall processing capability of the model during pre-training, training sets with high generality across multiple application scenarios are typically used. Therefore, the target annotation model obtained through pre-training is a general audio annotation model. Based on this, when using this general audio annotation model to annotate audio for certain specific application scenarios, the annotation effect may not be ideal. In this case, retraining the target annotation model would involve a large amount of data processing and a correspondingly high processing cost. Therefore, the model training method of this disclosure can be adopted, which utilizes audio training data for specific application scenarios and combines it with an audio recognition model to quickly improve the audio annotation quality of the target annotation model in that specific application scenario based on a self-supervised method.

[0043] In summary, in this embodiment of the present disclosure, the target annotation model can be quickly adjusted to be suitable for audio annotation in a specific application domain through multiple rounds of iterative training. Specifically, for any round, the first audio training data corresponding to the preset application domain is first input into the first target annotation model and the first audio recognition model of the current round to obtain the first predicted annotation data. Then, the first predicted annotation data is filtered according to the confidence level of the first predicted annotation data, and the part of the first audio training data with lower annotation quality (i.e., the target audio training data) is deleted to obtain the second audio training data with higher annotation quality. The first audio recognition model is first loaded and trained using the second audio training data to obtain the second audio recognition model. Then, the first target annotation model is trained using the second audio training data and the second audio recognition model to obtain the second target annotation model with better performance, and then enters the next round of training. Therefore, if the initially obtained target annotation model (i.e. the target annotation model that has not yet undergone the first round of processing) does not perform well in the preset application domain, the network parameters of the target annotation model can be quickly adjusted in the above manner, so that the trained target annotation model can be quickly applied to the aforementioned preset application domain, thereby enabling the target annotation model to be applied to audio annotation in the preset application domain and achieving a higher processing effect.

[0044] The model training method of this disclosure embodiment will be described in detail below.

[0045] In some optional embodiments, the first audio recognition model and the second audio recognition model include at least one of an audio annotation model, an emotion recognition model, and a language recognition model, and the prior information includes at least one of pseudo-label data, emotion recognition data, and language recognition data; wherein, when there are multiple audio annotation models, the pseudo-label data is sorted in a random manner or according to the confidence level of the pseudo-label data.

[0046] For example, the audio annotation model includes a character annotation model, which can yield pseudo-label data including pseudo-labels for characters.

[0047] For example, the emotion recognition model can obtain emotion recognition data corresponding to the audio training data, which may include data such as happy, angry, and neutral.

[0048] For example, language identification data can be obtained from the language identification data corresponding to the audio training data, thereby clarifying the language information corresponding to the audio.

[0049] For example, the audio recognition model may also include a dialect recognition model, which can identify the dialect information corresponding to the audio.

[0050] For example, the first audio recognition model includes audio recognition model 1, audio recognition model 2, and audio recognition model 3, which respectively obtain the confidence scores 1, 2, and 3 of pseudo-labels 1 and 2, and 3 of pseudo-labels 1 and 2, where confidence score 1 < confidence score 3 < confidence score 2. Based on this, if a random sorting method is used, the order of pseudo-labels 1 to 3 can be arbitrarily selected and input into the first target labeling model, and the model can be instructed to randomly sort the pseudo-labels. If sorting is done in descending order of confidence score, "pseudo-label 2, pseudo-label 3, and pseudo-label 1" can be input into the first target labeling model, and the model can be instructed to sort the pseudo-labels in descending order of confidence score. The confidence score sorting itself also serves as part of the prior information, making the first target labeling model more targeted when referencing prior information, which helps improve the model's learning ability during training and thus improves the model training effect.

[0051] It should be noted that the above audio recognition models are merely illustrative examples, and the embodiments disclosed herein are not intended to limit the scope of the invention.

[0052] In summary, audio recognition models include several open-source or self-developed small audio recognition models. Due to their small size and their specific suitability for processing audio modal data, these small audio recognition models can quickly obtain corresponding prior information when processing audio training data. This prior information can then be applied to the inference or training of the first target labeling model.

[0053] In some optional embodiments, the model training method may further include: pre-training an initial annotation model to obtain a target annotation model; if the annotation results of the target annotation model in a preset application domain do not meet preset evaluation conditions, the target annotation model is used as the first target annotation model to enter the first round; wherein, the initial annotation model consists of a first network structure and a second network structure connected in series, the first network structure is used to obtain audio features adapted to the second network structure based on the input audio modal data, and the second network structure is used to determine text output data corresponding to the audio modal data and including predicted annotations based on the audio features.

[0054] In some optional embodiments, the first network structure includes a serially connected encoding network module and an adaptation network module. The encoding network module is used to encode the input audio modal data to obtain a first audio feature, and the adaptation network module is used to adjust the first audio feature to a second audio feature that is adapted to the second network structure. The second network structure is a network structure built based on a Large Language Model (LLM).

[0055] For example, the first network structure includes a serially connected encoder and adapter. The encoder can encode the input audio modal data to obtain first audio embedding features. Compared with the audio modal data, the first audio features are higher-dimensional data. However, the dimension of the first audio features may not be consistent with the dimension of the input data supported by the second network structure. Therefore, the adapter can be used to adjust the dimension of the first audio features so that the dimension of the obtained second audio features is consistent with the dimension of the input data supported by the second network structure.

[0056] Therefore, the encoder can convert audio modal data into high-dimensional audio features that the model can understand, and the adapter can adopt a lightweight network structure to map the audio features to the text semantic space corresponding to the second network structure, so that the second network structure can understand the audio content.

[0057] It should be noted that in related technologies, audio modal data is typically first converted into text features or text data before being input into a second network structure for processing. This approach does not truly apply the text processing capabilities of the second network structure to audio data processing; it essentially still utilizes the second network structure's ability to process text modal data. In contrast, in this embodiment, the first network structure converts audio modal data into audio features and inputs them into the second network structure. The second network structure can directly understand and process these audio features, thereby outputting text-based output data including predicted annotations. This truly applies the second network structure's text processing capabilities to the processing of audio modal data.

[0058] In some optional embodiments, pre-training the initial annotation model to obtain the target annotation model includes: adjusting the first network parameters of the initial annotation model based on preset second audio training data to obtain an intermediate annotation model, wherein the first network parameters are the network parameters corresponding to the first network structure; and adjusting the second network parameters of the intermediate annotation model based on the second audio training data and prior information corresponding to the second audio training data to obtain the target annotation model, wherein the second network parameters are at least some of the network parameters corresponding to the second network structure. The second audio training data may include audio data from multiple application domains, used to enable the target representation model to learn a relatively broad range of audio annotation capabilities.

[0059] Therefore, corresponding to the structure of the initial labeled model, the model pre-training process can also be divided into two stages: In the first stage, the network parameters of the second network structure are fixed, and only the network parameters of the first network structure (i.e., the first network parameters) are adjusted. Based on this, on the one hand, the data processing capability of the second network structure is not affected, and on the other hand, the audio training data can be used to focus on adjusting the network parameters of the first network structure, thereby achieving effective training of the first network structure; In the second stage, the network parameters of the already trained first network structure can be fixed, and at least some of the network parameters of the second network structure (i.e., the second network parameters) can be adjusted according to the audio training data and its corresponding prior information, so as to achieve effective training of the second network structure and finally obtain the trained target labeled model.

[0060] In some optional embodiments, the second audio training data includes multiple audio sample data and real labeled data corresponding to each audio sample data; accordingly, based on the preset second audio training data, the first network parameters of the initial labeling model are adjusted to obtain an intermediate labeling model, including: inputting the audio sample data into the initial labeling model to obtain third predicted labeling data; determining a first loss value based on the real labeled data and the third predicted labeling data; adjusting the first network parameters based on the first loss value; and determining an intermediate labeling model based on the current first network parameters when a first preset condition is met.

[0061] In some optional embodiments, the second network parameters of the intermediate annotation model are adjusted based on the second audio training data and the prior information corresponding to the second audio training data to obtain the target annotation model, including: inputting audio sample data and corresponding prior information into the intermediate annotation model to obtain fourth predicted annotation data; determining a second loss value based on the real annotation data and the fourth predicted annotation data; adjusting the second network parameters based on the second loss value; and determining the target annotation model based on the current second network parameters when a second preset condition is met.

[0062] Therefore, the initial annotation model in this embodiment includes two parts: a first network structure and a second network structure, each with different functions to support different data processing capabilities. The first network structure can directly receive and process audio modal data. Furthermore, the first network structure has a certain network adaptation function, capable of adjusting the audio features extracted from the audio modal data to fit the second network structure, allowing the second network structure to further process the audio features. The second network structure itself possesses strong text processing capabilities. However, if it directly processes audio modal data, the processing effect is poor due to modal differences between audio and text data, making it impossible to effectively apply its text processing capabilities to audio modal data processing. However, the audio features obtained through the processing of audio modal data using the first network structure are compatible with the second network structure. Therefore, the second network structure can be used to further process the audio features, thereby applying its text processing capabilities to the audio data processing and achieving better processing results.

[0063] In other words, relying solely on audio processing models cannot directly correct auditory ambiguities caused by the audio data itself. However, by combining the first network structure with the second network structure, the text processing capabilities of the second network structure can be applied to the processing of audio data. The second network structure can be used as a semantic constraint for audio processing to correct auditory ambiguities caused by the audio data itself, thereby improving the accuracy of the audio processing results.

[0064] The following is combined Figure 2 and Figure 3 The pre-training process of the target annotation model in the embodiments of this disclosure is illustrated by way of example.

[0065] Figure 2 This is a schematic diagram illustrating a model training method provided in an embodiment of this disclosure. (Refer to...) Figure 2 The first network structure is serially connected to the second network structure, and the first network structure includes a serially connected encoder and adapter. The audio training data includes multiple audio sample data (wav) and corresponding ground truth annotation data for each audio sample data (wav). The ground truth annotation data may include corresponding ground truth text data (T_text) and ground truth labels (T_label).

[0066] like Figure 2 As shown, during the i-th round of training, the corresponding multiple audio sample data wav are input into the encoder. After the encoder processes the data, the initial audio features are obtained. The initial audio features are then input into the adapter, which adjusts the dimensions of the initial audio features to obtain audio features that are adapted to the input dimensions of the second network structure. The audio features are then input into the second network structure, where i is an integer greater than or equal to 1.

[0067] Further, a first preprocessing instruction to "recognize this audio" is input into the second network structure. The second network structure performs data processing such as audio recognition based on the input audio features and the first preprocessing instruction, thereby obtaining first predicted annotation data corresponding to the audio sample data wav_i. The first predicted annotation data may include the recognized text F_text_i and the predicted label F_label_i corresponding to the recognized text F_text_i. After obtaining the first predicted annotation data, the value of a preset first loss function can be determined by combining it with the true label F_label_i corresponding to the audio sample data wav_i, thus obtaining the first loss value.

[0068] If the first loss value is less than or equal to the first preset loss threshold, or the current training epoch i is greater than or equal to the first preset epoch threshold, then the first preset condition is met, training is stopped, and an intermediate labeled model is determined based on the current first network parameters.

[0069] Conversely, if the first loss value is greater than the first preset loss threshold, or the current training round i is less than the first preset round threshold, then it is determined that the first preset condition is not met, and the first network parameters corresponding to the first network structure are adjusted according to the first loss value before entering the (i+1)th round of training.

[0070] Figure 3 This is a schematic diagram illustrating a model training method provided in an embodiment of this disclosure. (Refer to...) Figure 3 The first network parameters of the first network structure have been trained and are kept unchanged during this training phase. Only some parameters of the second network structure are adjusted. The prior information includes pseudo-labels 1 to n obtained by audio recognition models 1 to n recognizing audio sample data.

[0071] like Figure 3 As shown, during the j-th round of training, the corresponding multiple audio sample data wav are input into the encoder. After the encoder processes the data, the initial audio features are obtained. The initial audio features are then input into the adapter, which adjusts the dimensions of the initial audio features to obtain audio features that are adapted to the input dimensions of the second network structure. The audio features are then input into the second network structure, where j is an integer greater than or equal to 1.

[0072] Furthermore, a second preprocessing instruction is input into the second network structure: "Recognize this audio. The following are possible contents: pseudo-label 1 labeled by preset audio recognition model 1; pseudo-label 2 labeled by preset audio recognition model 2; ... pseudo-label n labeled by preset audio recognition model n." The second network structure performs data processing such as audio recognition based on the input audio features and the second preprocessing instruction, thereby obtaining the second predicted label data corresponding to the audio sample data wav_j. The second predicted label data may include the recognized text F_text_j and the predicted label F_label_j corresponding to the recognized text F_text_j. After obtaining the second predicted label data, the value of the preset second loss function can be determined by combining it with the true label F_label_j corresponding to the audio sample data wav_j, thus obtaining the second loss value.

[0073] If the second loss value is less than or equal to the second preset loss threshold, or the current training epoch j is greater than or equal to the second preset epoch threshold, then the second preset condition is met, training is stopped, and the target labeled model is determined based on the current second network parameters.

[0074] Conversely, if the second loss value is greater than the second preset loss threshold, or the current training round j is less than the second preset round threshold, then it is determined that the second preset condition is not met, and the second network parameters are adjusted according to the second loss value before entering the (j+1)th round of training.

[0075] Furthermore, after obtaining the target annotation model through the above pre-training, if the annotation results of the target annotation model in the preset application domain do not meet the preset evaluation conditions, the target annotation model can be used as the first target annotation model to enter the first round, so as to train the first target annotation model and obtain an audio annotation model with a high annotation effect in the preset application domain.

[0076] As mentioned earlier, the processing flow for any given round includes the following: First, based on the first audio training data, the first audio recognition model and the first target annotation model are used to obtain the first predicted annotation data. Then, the target audio training data is removed from the first audio training data using the confidence level of the first preset annotation data. The remaining first audio training data is used as the second audio training data. Next, the first audio recognition model is trained using the second audio training data to obtain the second audio recognition model. Then, based on the second audio training data, the first target annotation model is trained using the second audio recognition model to obtain the second target annotation model. After completing the training for the current round, if the preset convergence condition is met, training stops, and the current second target annotation model is used as the final audio annotation model applicable to the preset application domain. If the preset convergence condition is not met, the current round's second target annotation model is used as the first target annotation model for the next round, and the current round's second audio recognition model is used as the first audio recognition model for the next round, thus entering the next round of training. The training process for the next round is similar to that of the current round and will not be described in detail here.

[0077] It should be noted that the first audio training data is audio training data related to a preset application domain. By training the model using the first audio training data, the model can learn the audio features of the preset application domain, thereby enabling high-quality annotation of the audio data of the preset application domain.

[0078] Furthermore, in any round of training, the first audio training data is first input into the first target labeling model and the first audio recognition model respectively. After the first audio recognition model performs audio recognition and other processing on the first audio training data, it inputs the obtained first prior information into the first target labeling model. The first target labeling model performs corresponding data processing based on the first audio training data and the first prior information to obtain the first predicted labeling data.

[0079] In some optional embodiments, the first audio training data corresponding to a preset application domain is input into the first target annotation model and the first audio recognition model of the current round to obtain the first predicted annotation data corresponding to the first audio training data. This includes: for the current round, inputting the first audio training data into the first audio recognition model to obtain the first prior information corresponding to the first audio training data; determining a first processing instruction based on the first prior information, the first processing instruction being used to instruct the processing method of the first target annotation model; and inputting the first audio training data and the first processing instruction into the first target annotation model to obtain the first predicted annotation data.

[0080] In some alternative implementations, the first processing instruction is generated based on first prior information and the desired reasoning method for the first target annotation model. This type of first processing instruction explicitly instructs the first target annotation model to perform a forward inference process. Alternatively, the first processing instruction is generated based on the first prior information, and this type of first processing instruction does not explicitly instruct the first target annotation model to perform a forward inference process. The desired reasoning method includes requirements for the reasoning logic and output data of the first target annotation model. The requirements for the reasoning logic reflect the processing logic that the first target annotation model needs to follow when performing forward inference, and the requirements for the output data reflect the content and format of the output data after the first target annotation model has completed forward inference.

[0081] In some alternative implementations, a first processing instruction can be generated based on the first prior information and the inference content expected to be performed by the first target annotation model (corresponding to the processing method of the first target annotation model). This first processing instruction is then input into the first target annotation model to instruct it to perform the corresponding data processing. In this approach, the processing method of the first target annotation model needs to be predetermined and explicitly input into the model to clearly guide the model to perform which data processing.

[0082] In some alternative implementations, a first processing instruction can be directly generated based on the first prior information and input into the first target annotation model. This allows the first target annotation model to use the first processing instruction as reference information to perform corresponding data processing. In this approach, it is not necessary to predefine the processing method of the first target annotation model, nor is it necessary to explicitly input the processing method into the model. Instead, the model itself determines which data processing should or can be performed based on the first prior information.

[0083] For example, the first processing instruction can take the form of a prompt message. Assuming the first prior information includes n pseudo-labels, the prompt message "Identify this audio; the following are possible contents: pseudo-label 1, pseudo-label 2, ..., pseudo-label n" can be input into the first target annotation model. Correspondingly, the first target annotation model performs corresponding data processing (i.e., identifies the audio and predicts the label) based on this prompt message and the first audio training data to obtain the first predicted annotation data. The first predicted annotation data includes the identified text, the predicted label corresponding to the identified text, and the confidence score of each predicted label.

[0084] Therefore, the core function of this type of first processing instruction is to guide the model to perform the corresponding data processing more accurately through a clear description of the requirements. In other words, the first processing instruction can more clearly and specifically instruct the model to perform the corresponding data processing, thereby improving the model's processing efficiency.

[0085] For example, assuming the first prior information includes n pseudo-labels, this first prior information can be directly used as the first processing instruction. A prompt message is then used to input "The following are possible contents: pseudo-label 1, pseudo-label 2, ..., pseudo-label n" into the first target annotation model. Upon receiving this prompt message, the first target annotation model can infer the predicted label for the audio to be processed and use "pseudo-label 1, pseudo-label 2, ..., pseudo-label n" as reference information to perform forward inference, obtaining the corresponding first predicted label data. In this process, the first processing instruction mainly serves as reference information, guiding the first target annotation model to perform forward inference, without explicitly specifying the model's processing method. Therefore, it effectively saves the operation of determining the model's processing method and explicitly inputting the processing method, simplifying the training process to some extent.

[0086] In some optional implementations, the first target labeling model is used to obtain the first predicted labeling data through at least one of the following processing methods: speech recognition, label prediction, and pseudo-label tracing.

[0087] For example, the first processing instruction is used to instruct the first target annotation model to perform at least one of speech recognition, label prediction, and pseudo-label tracing based on the first audio training data and the first prior information to obtain first predicted annotation data; the first predicted annotation data includes at least one of recognized text data, predicted labels corresponding to each recognized text data, and pseudo-label tracing prediction data; wherein, the pseudo-label tracing prediction data is used to characterize the first audio recognition model that generates pseudo-label data for each pseudo-label data prediction.

[0088] For example, the first audio training data includes multiple audio sample data and corresponding real annotation data (such as real labels) for each audio sample data. When the first processing instruction instructs the first target annotation model to perform speech recognition and label prediction, the first target annotation model can perform speech recognition on each audio sample data, predict labels based on the speech recognition results, thereby obtaining the predicted labels for each audio sample data, and also obtaining the prediction confidence of the predicted labels. Furthermore, when the first processing instruction instructs the first target annotation model to perform pseudo-label tracing, the first target annotation model can determine each pseudo-label data based on first prior information, and predict the first audio recognition model that generates each pseudo-label data, thereby obtaining pseudo-label tracing prediction data.

[0089] In some optional implementations, pseudo-label tracing can be performed based on model label mapping relationship and / or model identification information. The model label mapping relationship is used to characterize the mapping relationship between the first audio recognition model and the label data generated by the first audio recognition model. The model identification information is unique information such as model fingerprint and model watermark that are implicitly embedded by the first audio recognition model when generating pseudo-label data, which is used to identify or distinguish the model.

[0090] The first target labeling model can learn the mapping relationship between each first audio recognition model and the label data generated by the first audio recognition model. Based on this, when performing pseudo-label tracing, the first target labeling model can perform feature extraction and other processing (such as fingerprint extraction) on each pseudo-label data according to the learned mapping relationship, and perform model classification according to the extracted feature data to obtain the classification result corresponding to each pseudo-label data. Thus, the first audio recognition model indicated by the classification result is used as the predicted first audio recognition model that generates the pseudo-label data.

[0091] Furthermore, the first audio recognition model can implicitly embed unique, hidden model identification information such as model fingerprints and model watermarks when generating pseudo-label data. Based on this, after obtaining the pseudo-label data, the first target annotation model can extract the model identification information from the pseudo-label data, thereby tracing back to the first audio recognition model that generated the pseudo-label data through the model identification information. The model identification information extracted from the pseudo-label data can be encrypted or encoded, and the first target annotation model can decrypt or decode it to obtain the model identification information, thus tracing back to the first audio recognition model that generated the pseudo-label data.

[0092] Therefore, it can be seen that the first target labeling model can perform one or more of the following processing methods based on the first prior information: speech recognition, label prediction, and pseudo-label tracing, thereby meeting different usage requirements.

[0093] In other words, the first processing instruction can not only instruct the first target labeling model to recognize audio content and determine the corresponding predicted label, but also instruct it to predict the source of each pseudo-label (i.e., predict the first audio recognition model that generates the pseudo-label), and instruct it to refer to the confidence level of each pseudo-label during the forward inference process, thereby assigning different pseudo-labels different weights and performing audio recognition and label prediction operations based on different weights.

[0094] For example, the pseudo-label data output by the first audio recognition model can be randomly combined, allowing the first target annotation model to predict its source during processing. This increases the prior information of the first target annotation model regarding different input audio recognition models. For the first target annotation model, it needs to randomly select pseudo-labels corresponding to the same audio sample data (wav) from N audio recognition models and determine the first audio recognition model that generates each pseudo-label. For example, for {wav_i, label_i, psedo1_i, psedo2_i, psedo3_i}, the first target annotation model needs to predict the source of psedo1_i, psedo2_i, and psedo3_i while predicting the correct label (label_i). In this way, the first target annotation model can distinguish the source of each pseudo-label before predicting the correct recognized text, thus allowing it to focus on referring to pseudo-labels from one or more specific application domains.

[0095] For example, we can construct {wav_i,label_i,psedo1_i,psedo2_i,psedo3_i}, where the confidence of psesedo1_i, psesedo2_i, and psesedo3_i decreases sequentially. This allows the first target annotation model to selectively refer to psesedo1_i, psesedo2_i, and psesedo3_i, thereby learning audio annotation capabilities more effectively.

[0096] Furthermore, after obtaining the first predicted label data, there may be some predicted labels with low confidence. Since low confidence indicates that the accuracy of the corresponding predicted label may be low, the first audio training data can be filtered based on confidence. The first audio training data corresponding to the predicted labels with low confidence can be deleted, and the remaining first audio training data corresponding to the predicted labels with high confidence becomes the second audio training data. This filtering operation can improve the reliability of audio annotation.

[0097] In some optional embodiments, the first prediction annotation data includes at least one identified text data, a predicted label for each identified text data, and a confidence level for each predicted label; correspondingly, the target audio training data in the first audio training data is deleted according to the first prediction annotation data to obtain the second audio training data, including: selecting the second audio training data from the first audio training data according to the confidence level of the predicted label for each identified text data in the first prediction annotation data and preset filtering conditions.

[0098] In some optional embodiments, the preset filtering conditions include at least one of the following: deleting the first audio training data corresponding to the predicted labels whose confidence level is less than or equal to a preset confidence threshold; deleting the first audio training data corresponding to the predicted labels whose confidence level is ranked at a preset ranking ratio threshold.

[0099] For example, if the confidence threshold is set to 90%, the first audio training data corresponding to the predicted labels with a confidence level of less than or equal to 90% will be deleted, and the remaining data will be the second audio training data.

[0100] For example, if the preset sorting ratio threshold is 20%, then all predicted labels are arranged in order of confidence value from low to high to obtain a predicted label sequence. The first audio training data corresponding to the first 20% of the predicted labels in this predicted label sequence is deleted, and the remaining data is the second audio training data.

[0101] In some optional embodiments, the first audio training data includes multiple audio sample data wav and real labeled data corresponding to each audio sample data wav. The real labeled data may include the corresponding real text data T_text and real label T_label.

[0102] Audio sample data (wav) is input into the first network structure of the first target annotation model and each first audio recognition model. The first network structure encodes and adjusts each audio sample data (wav) to obtain audio features, which are then input into the second network structure. Each first audio recognition model performs audio recognition and other processing on each audio sample data (wav) to obtain corresponding first prior information. Based on the first prior information of all first audio recognition models, it determines processing instructions and inputs these instructions into the second network structure. The second network structure generates a first hidden vector according to the processing instructions, obtains a second hidden vector based on the audio features, and concatenates the first and second hidden vectors to obtain a fused hidden vector. This fused hidden vector is further processed before outputting the first predicted annotation data. For example, the first predicted annotation data includes...<start_fix> Recognize text 1<end_fix> label1, c1;<start_fix> Recognize text 2<end_fix> label2, c2; ...,<start_fix> Recognize text m<end_fix> labelm, cm; <eos>,in,<start_fix> Indicates the start symbol of the text.<end_fix> This indicates the end-of-text symbol. The recognized texts 1 to m are obtained through audio recognition. label1 to labelm are the predicted labels corresponding to recognized texts 1 to m, respectively. label21 to label2n are the predicted labels corresponding to recognized texts 2, ..., labelm1 to labelmn are the predicted labels corresponding to recognized texts m. c1 to cm are the confidence scores corresponding to the predicted labels labelm1 to labelmn. eos is the output termination symbol.

[0103] Furthermore, if the confidence threshold is preset to thr, then audio sample data wav and their real labeled data that are less than or equal to thr from c1 to cm can be deleted, and the remaining audio sample data and their real labeled data constitute the second audio training data.

[0104] Therefore, by performing a confidence screening operation, audio sample data with low confidence can be deleted, and only audio sample data with high reliability can be retained. Based on this, when using the screened audio sample data to train the model (including the first audio recognition model and the first target annotation model), the training effect of the model can be effectively improved.

[0105] As mentioned earlier, after obtaining the second audio training data with higher labeling reliability, the first audio recognition model and the first target labeling model can be trained sequentially using the second audio training data.

[0106] In some optional embodiments, the first audio recognition model can be trained using noise-injection training on the second audio training data to obtain a second audio recognition model with better processing performance. Noise-injection training mainly refers to introducing controllable noise into the second audio training data during the training process, forcing the first audio recognition model to learn the core features of the data rather than redundant information related to noise, ultimately improving the generalization ability, anti-interference ability, and robustness of the first audio recognition model.

[0107] For example, firstly, noise that fits the current training scenario (such as background noise or channel noise corresponding to audio) is designed, and the aforementioned noise is injected into the second audio training data according to a preset intensity and / or preset ratio. Then, the first audio recognition model is trained using the noise-added second audio training data, so that the first audio recognition model automatically filters noise and focuses on the core features of the audio during the learning process, and finally obtains the second audio recognition model.

[0108] Furthermore, the first target annotation model can be trained based on the second audio training data and using the second audio recognition model to obtain a second target annotation model with better annotation performance.

[0109] In some optional embodiments, training a first target annotation model based on second audio training data and a second audio recognition model to obtain a second target annotation model for the current round includes: inputting the second audio training data into the second audio recognition model to obtain second prior information corresponding to the second audio training data; determining a second processing instruction based on the second prior information, the second processing instruction being used to instruct the processing method of the second target annotation model; inputting the second audio training data and the second processing instruction into the second target annotation model to obtain second predicted annotation data; determining a loss value based on the second audio training data and the second predicted annotation data; and adjusting at least some network parameters of the second target annotation model based on the loss value to obtain the second target annotation model.

[0110] The network parameters involved in adjusting the second target annotation model may include the network parameters corresponding to the first network structure, and / or some network parameters corresponding to the second network structure, etc., and this disclosure does not limit this. Since the network parameters of the second network structure are relatively large, a low-rank adaptation (LORA) training method can be used to adjust its network parameters.

[0111] For example, the original weight matrix of the second network structure is denoted as W. 原始 The update amount of the weight matrix is ​​ΔW, and it satisfies W 新 =W 原始 +ΔW. Further, decompose ΔW into the product of two smaller matrices: Let A be a reduced-dimensional matrix and B be an increased-dimensional matrix, with the rank of both being less than the rank of the original weight matrix. During training, the parameters of matrices A and B can be adjusted to replace the full weight update with a minimal number of low-rank matrices (i.e., A and B), thus reducing training costs.

[0112] In summary, by training the first target annotation model with the second predicted annotation data, which has higher annotation reliability, and in combination with the second audio recognition model, which has better processing performance, the feature learning and processing capabilities of the first target annotation model for audio data in the preset application scenario can be effectively improved, thereby enhancing the processing capability of audio data in the preset application scenario.

[0113] In some optional implementations, the second target labeling model is used to obtain second predicted labeling data through at least one of the following processing methods: speech recognition, label prediction, label confidence prediction, and pseudo-label tracing.

[0114] In some optional embodiments, the second processing instruction is used to instruct the second target annotation model to perform at least one of speech recognition, label prediction, label confidence prediction, and pseudo-label tracing based on the second audio training data (i.e., audio modal data) and the second prior information (i.e., prior information corresponding to the audio modal data) to obtain second predicted annotation data; the second predicted annotation data includes at least one of at least one recognized text data, a predicted label corresponding to each recognized text data, a predicted confidence of each predicted label, and pseudo-label tracing prediction data; wherein, the pseudo-label tracing prediction data is used to characterize the audio recognition model that generates pseudo-label data for each pseudo-label data prediction.

[0115] Therefore, the second processing instruction can not only instruct the second target labeling model to recognize audio content and determine the corresponding predicted labels, but also instruct it to predict the source of each pseudo-label (i.e., predict the second audio recognition model that generates pseudo-labels), and instruct it to refer to the confidence of each pseudo-label during the forward inference process, thereby assigning different pseudo-labels different weights and performing audio recognition and label prediction operations based on different weights.

[0116] For example, the second audio training data includes multiple audio sample data and corresponding real annotation data (such as real labels) for each audio sample data. When the second processing instruction instructs the second target annotation model to perform speech recognition and label prediction, the second target annotation model can perform speech recognition on each audio sample data, predict labels based on the speech recognition results, thereby obtaining the predicted labels for each audio sample data, and also obtaining the prediction confidence of the predicted labels. Furthermore, when the second processing instruction instructs the second target annotation model to perform pseudo-label tracing, the second target annotation model can determine each pseudo-label data based on the second prior information, and predict the second audio recognition model that generates each pseudo-label data, thereby obtaining pseudo-label tracing prediction data.

[0117] In some optional implementations, pseudo-label tracing can be performed based on model label mapping relationship and / or model identification information. The model label mapping relationship is used to characterize the mapping relationship between the second audio recognition model and the label data generated by the second audio recognition model. The model identification information is unique information such as model fingerprint and model watermark that are implicitly embedded by the second audio recognition model when generating pseudo-label data, which is used to identify or distinguish the model.

[0118] The second target labeling model can learn the mapping relationship between each second audio recognition model and the label data generated by that second audio recognition model. Based on this, when performing pseudo-label tracing, the second target labeling model can perform feature extraction and other processing (such as fingerprint extraction) on each pseudo-label data according to the learned mapping relationship, and perform model classification according to the extracted feature data to obtain the classification result corresponding to each pseudo-label data. Thus, the second audio recognition model indicated by the classification result is used as the predicted second audio recognition model that generates the pseudo-label data.

[0119] Furthermore, the second audio recognition model can implicitly embed unique model identification information such as model fingerprints and model watermarks when generating pseudo-label data. Based on this, after obtaining the pseudo-label data, the second target annotation model can extract the model identification information from the pseudo-label data, thereby tracing back to the second audio recognition model that generated the pseudo-label data through the model identification information. The model identification information extracted from the pseudo-label data can be encrypted or encoded. The second target annotation model can first decrypt or decode this information to obtain the model identification information, and then trace back to the second audio recognition model that generated the pseudo-label data.

[0120] For example, the pseudo-label data output by the second audio recognition model can be randomly combined, allowing the first target annotation model to predict its source during processing. This increases the prior information of the first target annotation model regarding different input audio recognition models. For the first target annotation model, it needs to randomly select pseudo-labels corresponding to the same audio sample data (wav) from N audio recognition models and determine the second audio recognition model that generates each pseudo-label. For example, for {wav_i, label_i, psedo1_i, psedo2_i, psedo3_i}, the first target annotation model needs to predict the source of psedo1_i, psedo2_i, and psedo3_i while predicting the correct label (label_i). In this way, the first target annotation model can distinguish the source of each pseudo-label before predicting the correct recognized text, thus allowing it to focus on referring to pseudo-labels from one or more specific application domains.

[0121] For example, construct {wav_i,label_i,psedo1_i,psedo2_i,psedo3_i}, where the confidence of psesedo1_i, psesedo2_i, and psesedo3_i decreases sequentially. This allows the first target annotation model to selectively refer to psesedo1_i, psesedo2_i, and psesedo3_i, thereby learning audio annotation capabilities more effectively.

[0122] Figure 4 This is a schematic diagram illustrating a model training method provided in an embodiment of this disclosure. (Refer to...) Figure 4 In the first round, the first audio training data 1 is input into the first target annotation model 1_1, the first audio recognition model 1_1, and the first audio recognition model 2_1, respectively. The first audio recognition model 1_1 and the first audio recognition model 2_1 generate first prior information 1 based on the first audio training data 1, and input the first prior information 1 into the first target annotation model 1_1. The first target annotation model 1_1 performs forward inference based on the first audio training data 1 and the first prior information 1, outputs the first predicted annotation data 1, and filters the first audio training data 1 based on the first predicted annotation data 1 to obtain the second audio training data 1. The first audio recognition model 1_1 and the first audio recognition model 2_1 are trained using the second audio training data 1 to obtain the second audio recognition model 1_2 and the second audio recognition model 2_2. Based on this, the first target annotation model 1_1 is trained using the second audio training data 1 and the second audio recognition model 1_2 and the second audio recognition model 2_2 to obtain the second target annotation model 1_2.

[0123] If it is confirmed that the preset convergence condition is not met, the second target annotation model 1_2 will be used as the first target annotation model 1_2 in the second round, and the second audio recognition model 1_2 and the second audio recognition model 2_2 will be used as the first audio recognition model 1_2 and the first audio recognition model 2_2 in the second round, respectively.

[0124] In the second round, the first audio training data 2 is input into the first target annotation model 1_2, the first audio recognition model 1_2, and the first audio recognition model 2_2, respectively. The first audio recognition model 1_2 and the first audio recognition model 2_2 generate first prior information 2 based on the first audio training data 2, and input the first prior information 2 into the first target annotation model 1_2. The first target annotation model 1_2 performs forward inference based on the first audio training data 2 and the first prior information 2, outputs the first predicted annotation data 2, and filters the first audio training data 2 based on the first predicted annotation data 2 to obtain the second audio training data 2. The first audio recognition model 1_2 and the first audio recognition model 2_2 are trained using the second audio training data 2 to obtain the second audio recognition model 1_3 and the second audio recognition model 2_3. Based on this, the first target annotation model 1_2 is trained using the second audio training data 2 and the second audio recognition model 1_3 and the second audio recognition model 2_3 to obtain the second target annotation model 1_3.

[0125] If it is confirmed that the preset convergence condition is not met, the second target annotation model 1_3 will be used as the first target annotation model 1_3 in the third round, and the second audio recognition model 1_3 and the second audio recognition model 2_3 will be used as the first audio recognition model 1_3 and the first audio recognition model 2_3 in the third round, respectively.

[0126] Similarly, in the nth round, the first audio training data n is input into the first target annotation model 1_n, the first audio recognition model 1_n, and the first audio recognition model 2_n, respectively. The first audio recognition model 1_n and the first audio recognition model 2_n generate first prior information n based on the first audio training data n, and input the first prior information n into the first target annotation model 1_n. The first target annotation model 1_n performs forward inference based on the first audio training data n and the first prior information n, outputs the first predicted annotation data n, and filters the first audio training data n based on the first predicted annotation data n to obtain the second audio training data n. The first audio recognition model 1_n and the first audio recognition model 2_n are trained using the second audio training data n to obtain the second audio recognition model 1_n+1 and the second audio recognition model 2_n+1. Based on this, the first target annotation model 1_n is trained using the second audio training data n and the second audio recognition model 1_n+1 and the second audio recognition model 2_n+1 to obtain the second target annotation model 1_n+1.

[0127] If it is confirmed that the preset convergence condition is not met, the second target annotation model 1_n+1 will be used as the first target annotation model 1_n+1 in the (n+1)th round, and the second audio recognition model 1_n+1 and the second audio recognition model 2_n+1 will be used as the first audio recognition model 1_n+1 and the first audio recognition model 2_n+1 in the (n+1)th round, respectively.

[0128] If the preset convergence condition is met, training is stopped, and the second target annotation model 1_n+1 is used as the audio annotation model applicable to the preset application domain.

[0129] Secondly, embodiments of this disclosure provide an audio data processing method.

[0130] Figure 5 A flowchart illustrating an audio data processing method provided in this embodiment of the disclosure. (Refer to...) Figure 5 The audio data processing method includes:

[0131] Step S51: Input the audio data to be labeled into the target labeling model and at least one preset audio recognition model respectively to obtain the target prediction labeling data corresponding to the audio data to be labeled.

[0132] The target labeling model is obtained using the model training method described in any one of the embodiments of this disclosure. The audio data to be labeled belongs to the audio data of a preset application domain. The audio recognition model is used to obtain prior information corresponding to the audio data to be labeled and input the prior information into the target labeling model so that the target labeling model can obtain target prediction labeling data based on the audio data to be labeled and the prior information.

[0133] The following is combined Figure 6 The audio data processing method of the present disclosure will be described in detail below.

[0134] Figure 6 This is a schematic diagram illustrating an audio data processing method provided in an embodiment of this disclosure. (Refer to...) Figure 6 The target annotation model includes a first network structure and a second network structure connected in series, and the first network structure further includes an encoder and an adapter connected in series.

[0135] like Figure 6 As shown, audio recognition models 1 to m are pre-set. When processing audio data from a preset application domain, the audio data to be labeled is input into the encoder of the first network structure and audio recognition models 1 to m, respectively. On one hand, the encoder encodes the audio data to be labeled to obtain initial audio features, which are then input into an adapter. The adapter adjusts the dimensions of these initial audio features to obtain audio features that match the input dimensions of the second network structure, and then inputs these audio features into the second network structure. On the other hand, any audio recognition model processes the input audio data to be labeled and outputs corresponding prior information, which is then input into the second network structure. The second network structure performs data processing such as audio recognition and audio labeling based on the input audio features and prior information, thereby obtaining the corresponding target prediction and labeling data.

[0136] In order to make the target labeling model perform inference tasks more conveniently, a third processing instruction can be input to it, such as "Please process the audio data to be labeled in combination with prior information, which includes prior information 1 output by audio recognition model 1, ..., prior information n output by audio recognition model n", or "Please process the audio data to be labeled in combination with prior information and predict the audio recognition model corresponding to each prior information, which includes prior information 1 output by audio recognition model 1, ..., prior information n output by audio recognition model n", or "Please process the audio data to be labeled in combination with prior information, which includes prior information 1 output by audio recognition model 1, ..., prior information n output by audio recognition model n, and the confidence of prior information 1 to prior information n decreases sequentially".

[0137] In summary, in the embodiments of this disclosure, the target annotation model obtained by the model training method of this disclosure has a good processing effect when processing data in the preset application domain. Furthermore, since the target annotation model effectively combines the first network structure and the second network structure, it can effectively utilize the powerful modeling and text processing capabilities of the second network structure itself to achieve accurate processing of audio modal data, thereby obtaining highly accurate processing results. In addition, by using the processing capabilities of several audio recognition models to inject prior information into the target annotation model as reference information, it can guide and promote the model to perform corresponding data processing efficiently and accurately, thereby improving processing efficiency and accuracy.

[0138] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0139] In addition, this disclosure also provides a model training device, an audio data processing device, an electronic device, a computer-readable storage medium, and a computer program product, all of which can be used to implement any of the model training methods or audio data processing methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0140] Figure 7 This is a block diagram of a model training apparatus provided in an embodiment of the present disclosure.

[0141] Reference Figure 7 This disclosure provides a model training apparatus 700, which includes:

[0142] The prediction module 701 is used to input the first audio training data corresponding to the preset application domain into the first target annotation model and the first audio recognition model of the current round for any round, and obtain the first prediction annotation data corresponding to the first audio training data. The first audio recognition model is used to obtain the prior information corresponding to the first audio training data, and the number of the first audio recognition models is one or more.

[0143] The deletion module 702 is used to delete the target audio training data in the first audio training data according to the first predicted annotation data to obtain the second audio training data. The confidence of the first predicted annotation data corresponding to the target audio training data is less than the confidence of the first predicted annotation data corresponding to the second audio training data.

[0144] The first training module 703 is used to perform noise-added training on the first audio recognition model based on the second audio training data to obtain the second audio recognition model.

[0145] The second training module 704 is used to train the first target annotation model based on the second audio training data and the second audio recognition model to obtain the second target annotation model for the current round.

[0146] The loop module 705 is used to execute the next round of model training when it is determined that the preset convergence condition is not met, wherein the second target annotation model and the second audio recognition model in the current round are respectively used as the first target annotation model and the first audio recognition model in the next round.

[0147] Therefore, in this embodiment of the present disclosure, the target annotation model can be quickly adjusted to be suitable for audio annotation in a specific application domain through multiple rounds of iterative training. Specifically, for any round, the first audio training data corresponding to the preset application domain is first input into the first target annotation model and the first audio recognition model of the current round to obtain the first predicted annotation data. Then, the first predicted annotation data is filtered according to the confidence level of the first predicted annotation data, and the part of the first audio training data with lower annotation quality (i.e., the target audio training data) is deleted to obtain the second audio training data with higher annotation quality. The first audio recognition model is then trained with noise added using the second audio training data to obtain the second audio recognition model. After obtaining the second audio recognition model, the first target annotation model is then trained using the second audio training data and the second audio recognition model to obtain the second target annotation model with better performance, and then enters the next round of training. Therefore, if the initially obtained target annotation model (i.e. the target annotation model that has not yet undergone the first round of processing) does not perform well in the preset application domain, the network parameters of the target annotation model can be quickly adjusted in the above manner, so that the trained target annotation model can be quickly applied to the aforementioned preset application domain, thereby enabling the target annotation model to be applied to audio annotation in the preset application domain and achieving a higher processing effect.

[0148] Figure 8 This is a block diagram of an audio data processing apparatus provided in an embodiment of the present disclosure.

[0149] Reference Figure 8 This disclosure provides an audio data processing apparatus 800, which includes:

[0150] The processing module 801 is used to input the audio data to be labeled into the target labeling model and at least one preset audio recognition model respectively, so as to obtain the target prediction labeling data corresponding to the audio data to be labeled;

[0151] The target labeling model is obtained using the model training method described in any one of the embodiments of this disclosure. The audio data to be labeled belongs to the audio data of a preset application domain. The audio recognition model is used to obtain prior information corresponding to the audio data to be labeled and input the prior information into the target labeling model so that the target labeling model can obtain target prediction labeling data based on the audio data to be labeled and the prior information.

[0152] In summary, in the embodiments of this disclosure, the target annotation model obtained by the model training method of this disclosure has a good processing effect when processing data in the preset application domain. Furthermore, since the target annotation model effectively combines the first network structure and the second network structure, it can effectively utilize the powerful modeling and text processing capabilities of the second network structure itself to achieve accurate processing of audio modal data, thereby obtaining highly accurate processing results. In addition, by using the processing capabilities of several audio recognition models to inject prior information into the target annotation model as reference information, it can guide and promote the model to perform corresponding data processing efficiently and accurately, thereby improving processing efficiency and accuracy.

[0153] Each module in the aforementioned model training device and audio data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0154] Figure 9 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0155] Reference Figure 9 This disclosure provides an electronic device, which includes: at least one processor 901, at least one memory 902, and one or more I / O interfaces 903; wherein, the memory 902 stores one or more computer programs that can be executed by at least one processor 901, and the one or more computer programs are executed by at least one processor 901 to enable at least one processor 901 to perform the above-described model training method or audio data processing method.

[0156] Figure 10 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0157] Reference Figure 10 This disclosure provides an electronic device that includes multiple processing cores 1001 and an on-chip network 1002. The multiple processing cores 1001 are all connected to the on-chip network 1002, and the on-chip network 1002 is used to exchange data between the multiple processing cores and external data.

[0158] One or more processing cores 1001 store one or more instructions, and the one or more instructions are executed by one or more processing cores 1001 to enable one or more processing cores 1001 to perform the above-mentioned model training method or audio data processing method.

[0159] The modules in the aforementioned electronic devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0160] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the model training method or audio data processing method described above. The computer-readable storage medium may be volatile or non-volatile.

[0161] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described model training method or audio data processing method.

[0162] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0163] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0164] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0165] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0166] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0167] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0168] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0169] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0170] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0171] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.< / eos>

Claims

1. A model training method, characterized in that, include: For any round, the first audio training data corresponding to the preset application domain is input into the first target annotation model and the first audio recognition model of the current round to obtain the first predicted annotation data corresponding to the first audio training data. The first audio recognition model is used to obtain the prior information corresponding to the first audio training data, and the number of the first audio recognition models is one or more. The target audio training data in the first audio training data is deleted based on the first predicted annotation data to obtain the second audio training data. The confidence level of the first predicted annotation data corresponding to the target audio training data is less than the confidence level of the first predicted annotation data corresponding to the second audio training data. The first audio recognition model is trained with noise added based on the second audio training data to obtain the second audio recognition model. The first target annotation model is trained based on the second audio training data and the second audio recognition model to obtain the second target annotation model for the current round; If the preset convergence condition is not met, the next round of model training is performed, wherein the second target annotation model and the second audio recognition model of the current round are respectively used as the first target annotation model and the first audio recognition model of the next round.

2. The method according to claim 1, characterized in that, The first audio recognition model and the second audio recognition model include at least one of an audio annotation model, an emotion recognition model, and a language recognition model, and the prior information includes at least one of pseudo-label data, emotion recognition data, and language recognition data; In cases where there are multiple audio annotation models, the pseudo-label data is sorted randomly or according to the confidence level of the pseudo-label data.

3. The method according to claim 1, characterized in that, The step of inputting the first audio training data corresponding to the preset application domain into the first target annotation model and the first audio recognition model of the current round to obtain the first predicted annotation data corresponding to the first audio training data includes: For the current round, the first audio training data is input into the first audio recognition model to obtain the first prior information corresponding to the first audio training data; A first processing instruction is determined based on the first prior information, and the first processing instruction is used to indicate the processing method of the first target annotation model; The first audio training data and the first processing instruction are input into the first target annotation model to obtain the first predicted annotation data.

4. The method according to claim 3, characterized in that, The first processing instruction is used to instruct the first target annotation model to perform at least one of speech recognition, label prediction, and pseudo-label tracing based on the first audio training data and the first prior information to obtain the first predicted annotation data; The first prediction annotation data includes at least one of the following: identified text data, prediction labels corresponding to each identified text data, and pseudo-label traceability prediction data; The first prior information includes pseudo-label data, and the pseudo-label traceability prediction data is used to characterize the first audio recognition model that generates the pseudo-label data for each of the pseudo-label data.

5. The method according to claim 1, characterized in that, The step of training the first target annotation model based on the second audio training data and the second audio recognition model to obtain the second target annotation model for the current round includes: The second audio training data is input into the second audio recognition model to obtain the second prior information corresponding to the second audio training data. A second processing instruction is determined based on the second prior information, and the second processing instruction is used to indicate the processing method of the second target annotation model; The second audio training data and the second processing instructions are input into the second target annotation model to obtain the second predicted annotation data; The loss value is determined based on the second audio training data and the second prediction annotation data; The second target labeling model is obtained by adjusting at least some of the network parameters of the second target labeling model based on the loss value.

6. The method according to claim 5, characterized in that, The second processing instruction is used to instruct the second target labeling model to perform at least one of speech recognition, label prediction, label confidence prediction, and pseudo-label tracing based on the second audio training data and the second prior information, so as to obtain the second predicted labeling data; The second prediction labeling data includes at least one of the following: identified text data, prediction labels corresponding to each identified text data, prediction confidence of each prediction label, and pseudo-label traceability prediction data; The second prior information includes pseudo-label data, and the pseudo-label traceability prediction data is used to characterize the audio recognition model that generates the pseudo-label data, predicted for each of the pseudo-label data.

7. The method according to claim 1, characterized in that, The first prediction annotation data includes at least one identified text data, a prediction label for each identified text data, and a confidence level for each prediction label; The step of deleting the target audio training data from the first audio training data based on the first predicted annotation data to obtain the second audio training data includes: Based on the confidence level of the predicted label of each identified text data in the first predicted label data and the preset filtering conditions, the second audio training data is filtered out from the first audio training data.

8. The method according to claim 1, characterized in that, The method further includes: If the preset convergence condition is met, the current second target annotation model is used as the audio annotation model applicable to the preset application domain.

9. The method according to any one of claims 1 to 8, characterized in that, Prior to the first round, the method also includes: Pre-train the initial annotation model to obtain the target annotation model; If the annotation result of the target annotation model in the preset application field does not meet the preset evaluation conditions, the target annotation model will be used as the first target annotation model to enter the first round. The initial annotation model consists of a first network structure and a second network structure connected in series. The first network structure is used to obtain audio features adapted to the second network structure based on the input audio modal data. The second network structure is used to determine text output data corresponding to the audio modal data and including predicted annotations based on the audio features.

10. The method according to claim 9, characterized in that, The first network structure includes a serially connected encoding network module and an adaptation network module. The encoding network module is used to encode the input audio modal data to obtain a first audio feature, and the adaptation network module is used to adjust the first audio feature to a second audio feature that is compatible with the second network structure. The second network structure is a network structure built based on a large language model.

11. The method according to claim 10, characterized in that, The step of pre-training the initial annotation model to obtain the target annotation model includes: Based on the preset second audio training data, the first network parameters of the initial annotation model are adjusted to obtain the intermediate annotation model, where the first network parameters are the network parameters corresponding to the first network structure. Based on the second audio training data and the prior information corresponding to the second audio training data, the second network parameters of the intermediate annotation model are adjusted to obtain the target annotation model, wherein the second network parameters are at least some of the network parameters corresponding to the second network structure.

12. An audio data processing method, characterized in that, include: The audio data to be labeled is input into the target labeling model and at least one preset audio recognition model to obtain the target prediction labeling data corresponding to the audio data to be labeled. The target labeling model is obtained using the model training method described in any one of claims 1 to 10. The audio data to be labeled belongs to the audio data of a preset application domain. The audio recognition model is used to obtain prior information corresponding to the audio data to be labeled and input the prior information into the target labeling model so that the target labeling model can obtain the target predicted labeling data based on the audio data to be labeled and the prior information.

13. A model training device, characterized in that, include: The prediction module is used to input the first audio training data corresponding to the preset application domain into the first target annotation model and the first audio recognition model of the current round for any round, and obtain the first prediction annotation data corresponding to the first audio training data. The first audio recognition model is used to obtain the prior information corresponding to the first audio training data, and the number of the first audio recognition models is one or more. The deletion module is used to delete the target audio training data in the first audio training data according to the first predicted annotation data to obtain the second audio training data, wherein the confidence of the first predicted annotation data corresponding to the target audio training data is less than the confidence of the first predicted annotation data corresponding to the second audio training data. The first training module is used to perform noise-added training on the first audio recognition model based on the second audio training data to obtain the second audio recognition model. The second training module is used to train the first target annotation model based on the second audio training data and the second audio recognition model to obtain the second target annotation model for the current round. The loop module is used to execute the next round of model training if the preset convergence condition is not met, wherein the second target annotation model and the second audio recognition model of the current round are respectively used as the first target annotation model and the first audio recognition model of the next round.

14. An audio data processing apparatus, characterized in that, include: The processing module is used to input the audio data to be labeled into the target labeling model and at least one preset audio recognition model respectively, and obtain the target prediction labeling data corresponding to the audio data to be labeled; The target labeling model is obtained using the model training method described in any one of claims 1 to 11. The audio data to be labeled belongs to the audio data of a preset application domain. The audio recognition model is used to obtain prior information corresponding to the audio data to be labeled and input the prior information into the target labeling model so that the target labeling model obtains the target predicted labeling data based on the audio data to be labeled and the prior information.

15. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the model training method as described in any one of claims 1-11, or the audio data processing method as described in claim 12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method as described in any one of claims 1-10, or the audio data processing method as described in claim 12.

17. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the model training method as described in any one of claims 1-10, or the audio data processing method as described in claim 12.

Images