Audio perception large model training method and audio processing method

Abstract

This invention provides a method for training a large-scale audio perception model and an audio processing method, relating to the field of natural language processing technology. Through cross-scene audio representation comparison learning, it not only enhances the coverage and memorization ability of knowledge in the target scene, but also improves the ability to perform mixed processing and analysis between different tasks in the target scene by combining training with first audio samples from the target scene. Furthermore, introducing text attribute information and combining it with audio-text representation comparison learning within the same scene can increase the probability of expressing the same content with different audio in the target scene, such as different animal cries of fear. Transfer learning between semantically similar audio samples alleviates the problem of scarce low-resource data. Moreover, completing the training of the first candidate large-scale audio model can enhance the general ability of the second candidate large-scale audio model to extract audio representation vectors in the target scene, reducing the repetitiveness of subsequent training of the initial large-scale audio perception model.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for training a large audio perception model and an audio processing method. Background Technology

[0002] Audio perception is an intelligent processing technology that focuses on non-human voice signals such as ambient sound, music, industrial sounds, and biological sounds. By analyzing audio signals, it can achieve perception and understanding, feature extraction, and decision support for complex audio scenes, and is gradually becoming a core supporting technology in fields such as industrial operation and maintenance, smart homes, and public safety.

[0003] The existing training schemes for audio perception models mainly combine self-supervised pre-training and downstream task fine-tuning for a specific scenario. The core of this approach is to extend the mask autoencoder from the image domain to audio spectrogram processing, learn audio features through mask prediction criteria, and then use the audio features in downstream tasks.

[0004] However, existing training schemes for audio perception models often rely solely on audio data from a specific scenario for training. This presents significant challenges for scenarios where audio data is scarce. Furthermore, training with a mix of audio data from all scenarios can lead to the model's inability to effectively memorize and cover all knowledge, and similar audio from different scenarios can cause more severe hallucinations, resulting in significant model confusion. In addition, audio data lacks a clear semantic structure and contextual logic, making it difficult to learn more generalizable representations using only masking prediction criteria. Summary of the Invention

[0005] This invention provides a method for training a large audio perception model and an audio processing method to address the deficiencies in related technologies.

[0006] This invention provides a method for training a large audio perception model, comprising:

[0007] Acquire the first audio sample in each scenario, the text attribute information of the first audio sample, and the second audio sample with the task label of the perception task in the target scenario;

[0008] Based on the first audio sample and the text attribute information, a mask prediction criterion is applied, and audio representation comparison learning across scenes and audio-text representation comparison learning within the same scene are combined to train the first candidate audio model in the target scene to obtain the second candidate audio model.

[0009] Based on the perception task and the second alternative audio model, an initial audio perception model is constructed. Based on the second audio sample, the initial audio perception model is trained to obtain the target audio perception model.

[0010] According to the present invention, a method for training a large audio perception model is provided, wherein based on the first audio sample and the text attribute information, a mask prediction criterion is applied, and cross-scene audio representation comparison learning and same-scene audio-text representation comparison learning are combined to train a first candidate large audio model in the target scene to obtain a second candidate large audio model, the method includes the following steps:

[0011] Obtain third-party audio samples for various scenarios;

[0012] Based on the third audio sample, the mask prediction criterion is applied to train the general audio model to obtain the first candidate audio model.

[0013] According to the present invention, a method for training a large-scale audio perception model is provided, wherein the number of third audio samples includes multiple samples; the step of constructing an initial large-scale audio perception model based on the perception task and the second candidate large-scale audio model includes:

[0014] For any third audio sample, the third audio sample is segmented to obtain several audio segments, and based on the second candidate audio large model, the first audio representation vector of each audio segment and the second audio representation vector of the third audio sample are extracted.

[0015] Based on the similarity between each of the first audio representation vectors and the second audio representation vectors, each of the third audio samples is screened to determine the fourth audio sample, and based on the fourth audio sample, the first candidate audio large model is trained to obtain the third candidate audio large model.

[0016] Based on the first audio sample and the text attribute information, a mask prediction criterion is applied, and audio representation comparison learning across scenes and audio-text representation comparison learning within the same scene are combined to train the third alternative audio model in the target scene to obtain a fourth alternative audio model. Based on the perception task and the fourth alternative audio model, the initial audio perception model is constructed.

[0017] According to the present invention, a method for training a large audio perception model, wherein the step of filtering each third audio sample based on the similarity between each first audio representation vector and the second audio representation vector to determine a fourth audio sample includes:

[0018] Determine the proportion of each audio segment in each of the third audio samples whose similarity is greater than or equal to a first specified threshold;

[0019] Based on the proportion corresponding to each of the third audio samples, the third audio samples are filtered to determine the fourth audio sample.

[0020] According to the present invention, a method for training a large audio perception model, wherein the step of filtering each of the third audio samples based on the proportion corresponding to each of the third audio samples to determine the fourth audio sample includes:

[0021] A fifth audio sample is determined based on the proportion of each of the third audio samples that is greater than or equal to a second specified threshold, and a fourth audio sample is determined based on each of the audio segments in the fifth audio sample whose similarity is greater than or equal to the first specified threshold.

[0022] According to the audio perception large model training method provided by the present invention, the length of each audio segment in the same third audio sample is the same, and the length of each audio segment in different third audio samples is randomly determined within a preset length.

[0023] According to the audio perception large model training method provided by the present invention, the first objective of the audio representation comparison learning includes that the similarity of audio representation vectors in the same scene is higher than that of audio representation vectors in different scenes.

[0024] The second objective of the audio-text representation contrastive learning is that the similarity between the audio representation vector and the corresponding text attribute information text representation vector in the same scene is higher than the similarity between the audio representation vector and the text attribute information text representation vector.

[0025] According to the present invention, a method for training a large-scale audio perception model includes, based on the first audio sample and the text attribute information, applying a mask prediction criterion, and combining cross-scene audio representation comparison learning and same-scene audio-text representation comparison learning, training a first candidate large-scale audio model in the target scene to obtain a second candidate large-scale audio model, comprising:

[0026] The time-frequency features of the first audio sample are determined, and the time-frequency features are masked to obtain masked features. The masked features are then input into the first candidate audio large model to obtain the audio representation vector output by the first candidate audio large model.

[0027] Based on the audio representation vector, calculate the audio representation contrast loss corresponding to the first target in the target scene, and extract the text representation vector of the text attribute information. Based on the audio representation vector and the text representation vector, calculate the audio-text representation contrast loss corresponding to the second target in the target scene.

[0028] Based on the audio representation contrast loss and the audio text representation contrast loss, a first training loss is calculated, and based on the first training loss, the first candidate audio large model is trained to obtain the second candidate audio large model.

[0029] According to the present invention, a method for training a large audio perception model, wherein constructing an initial large audio perception model based on the perception task and a second candidate large audio model includes:

[0030] Based on the aforementioned perception task, determine the initial perception task header;

[0031] Based on the initial perception task head and the second alternative audio large model, the initial audio perception large model is constructed.

[0032] The present invention also provides an audio processing method, comprising:

[0033] Acquire the audio data to be processed for the perception task in the target scene;

[0034] The audio data to be processed is input into the target audio perception model to obtain the target perception result of the perception task output by the target audio perception model.

[0035] The target audio perception big model is trained based on the audio perception big model training method provided in the above embodiments.

[0036] The present invention also provides an audio perception large model training device, comprising:

[0037] The sample acquisition module is used to acquire the first audio sample in each scenario, the text attribute information of the first audio sample, and the second audio sample with the task label of the perception task in the target scenario.

[0038] The first training module is used to train the first candidate audio model in the target scene based on the first audio sample and the text attribute information, by applying the mask prediction criterion and combining cross-scene audio representation comparison learning and audio-text representation comparison learning in the same scene, to obtain the second candidate audio model.

[0039] The second training module is used to construct an initial audio perception model based on the perception task and the second alternative audio model, and to train the initial audio perception model based on the second audio sample to obtain the target audio perception model.

[0040] The present invention also provides an audio processing apparatus, comprising:

[0041] The data acquisition module is used to acquire the audio data to be processed;

[0042] An audio processing module is used to input the audio data to be processed into a target audio perception model and obtain the processing result output by the target audio perception model.

[0043] The target audio perception big model is trained based on the audio perception big model training method provided in the above embodiments.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the audio perception large model training method or the audio processing method described above.

[0045] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio perception large model training method or the audio processing method as described above.

[0046] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the audio perception large model training method or the audio processing method as described above.

[0047] The audio perception large-scale model training method and audio processing method provided by this invention utilize first audio samples and text attribute information, apply masking prediction criteria, and combine cross-scene audio representation comparison learning and same-scene audio-text representation comparison learning. Under the target scene, a first candidate audio large-scale model is trained to obtain a second candidate audio large-scale model. This not only fully absorbs the data knowledge of the first audio samples in various scenes, improving the general audio representation ability of the second candidate audio large-scale model in the target scene and exhibiting good generalization, but also effectively avoids knowledge confusion in different scenes, enhancing the audio perception ability of the perception task in the target scene. Through cross-scene audio representation comparison learning, not only can the coverage and memory ability of knowledge in the target scene be strengthened, but also the ability to perform mixed processing and analysis between different tasks in the target scene can be improved by combining first audio samples in the target scene with mixed training. Furthermore, introducing text attribute information and combining it with same-scene audio-text representation comparison learning can increase the probability of the same content being expressed using different audio in the target scene, such as different animal cries of fear. Transfer learning between semantically similar audio can alleviate the problem of low-resource data scarcity. Moreover, completing the training of the first candidate audio model can enhance the general ability of the second candidate audio model to extract audio representation vectors in the target scene, and reduce the repetitiveness of subsequent training of the initial audio perception model. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is one of the flowcharts illustrating the large-scale audio perception model training method provided by this invention.

[0050] Figure 2 This is the second flowchart of the audio perception large model training method provided by the present invention.

[0051] Figure 3 This is a flowchart illustrating the audio processing method provided by the present invention.

[0052] Figure 4 This is a schematic diagram of the structure of the audio perception large model training device provided by the present invention.

[0053] Figure 5 This is a schematic diagram of the audio processing device provided by the present invention.

[0054] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] Existing audio perception models rely on training data from only a specific scenario, making training challenging in scenarios with scarce audio data. Even when training with audio data from all scenarios, the model struggles to effectively memorize and cover all knowledge, and similar audio from different scenarios can lead to more severe hallucinations and model confusion. Furthermore, the lack of clear semantic structure and contextual logic in audio data makes it difficult to learn more generalizable representations using only masking prediction criteria.

[0057] Based on this, this embodiment of the invention provides a method for training a large audio perception model. For example... Figure 1 As shown, the method includes:

[0058] S11, acquire the first audio sample in each scene, the text attribute information of the first audio sample, and the second audio sample with the task label of the perception task in the target scene;

[0059] S12, based on the first audio sample and the text attribute information, apply the mask prediction criterion, combine cross-scene audio representation comparison learning and same-scene audio text representation comparison learning, train the first candidate audio large model in the target scene, and obtain the second candidate audio large model.

[0060] S13. Based on the perception task and the second alternative audio model, an initial audio perception model is constructed. Based on the second audio sample, the initial audio perception model is trained to obtain the target audio perception model.

[0061] Specifically, the audio perception large model training method provided in this embodiment of the invention is executed by an audio perception large model training device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0062] First, step S11 is executed to obtain the first audio sample for each scene, the text attribute information of the first audio sample, and the second audio sample with the task label of the perception task in the target scene. The scenes can include music scenes, industrial scenes, marine scenes, and animal scenes, etc. The first audio sample can include industrial machine audio, underwater marine audio, animal call audio, etc. The first audio sample can be used for unsupervised training without task labels, or for weakly supervised training with a small number of task labels; no specific limitation is made here.

[0063] The text attribute information of the first audio sample refers to the text information used to describe the attributes of the first audio sample. For example, it may include audio of transformer sound diagnosis in an industrial scenario, audio of cruise ship propeller in an ocean scenario, and audio of rock music in a music scenario.

[0064] The target scenario refers to the scenario in which downstream sensing tasks need to be performed. It can be any of the various scenarios, such as an industrial scenario. The sensing task can be a specific task within the target scenario, such as a prediction task, a reconstruction task, a classification task, etc. For example, it can be a lifespan prediction task, a voiceprint quality inspection task, a fault detection task, or a fault classification task in an industrial scenario.

[0065] The task label refers to the perception result corresponding to the perception task in the target scene, such as the fault classification result in an industrial scene. The second audio sample refers to the audio sample in the target scene that contains the perception result corresponding to the perception task. For example, it could be machine audio from a fault classification task in an industrial scene.

[0066] Then, step S12 is executed. Using the first audio sample and text attribute information, a mask prediction criterion is applied, combined with cross-scene audio representation comparison learning and same-scene audio-text representation comparison learning. Under the target scene, the first candidate audio model is trained to obtain the second candidate audio model. This first candidate audio model can be a general audio model or obtained through unsupervised training of a general audio model; no specific limitation is made here. The structure of the first candidate audio model can be a neural network, including but not limited to Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Transformers.

[0067] Understandably, the masking prediction criterion involves partially masking the time-frequency features of each first audio sample proportionally, i.e., obscuring some of the time-frequency features. The partially masked time-frequency features are then input into a first candidate audio model. This first candidate audio model predicts the masked audio representation vector using the unmasked portion of the audio representation vector, thus obtaining the audio representation vector for each first audio sample. Here, the time-frequency features can be Short-Time Fourier Transform (STFT) features or other relevant features representing time and frequency.

[0068] To eliminate the impact of differences in scene audio background on audio representation vectors, cross-scene audio representation contrastive learning is introduced to improve the similarity of audio representation vectors within the same scene. Cross-scene audio representation contrastive learning refers to calculating the audio representation contrastive loss by using the similarity of audio representation vectors of different first audio samples within the same scene and the similarity of audio representation vectors of first audio samples in different scenes.

[0069] To eliminate the influence of scene-related text attribute information on audio representation vectors, a scene-based audio-text representation contrastive learning approach is introduced to improve the similarity between the audio representation vector of the first audio sample and the text representation vector of its corresponding text attribute information within the same scene. Scene-based audio-text representation contrastive learning involves calculating the audio-text representation contrastive loss by using the similarity between the audio representation vector of the first audio sample and the text representation vector of its corresponding text attribute information within the same scene, as well as the similarity between the audio representation vector of the first audio sample and the text representation vectors of other text attribute information.

[0070] Here, the text representation vector can be extracted using a large text model. This involves inputting text attribute information into the large text model, which then extracts the text representation vector from the text attribute information. The large text model used in this embodiment can be selected as needed, and may include, for example, the QWen model.

[0071] In the target scene, the first candidate large audio model is trained. Specifically, in cross-scene audio representation contrastive learning, the audio representation contrastive loss is calculated using the similarity of audio representation vectors of different first audio samples in the target scene, as well as the similarity of audio representation vectors of first audio samples in the target scene with those in other scenes. In the same-scene audio-text representation contrastive learning, the audio-text representation contrastive loss is calculated using the similarity between the audio representation vector of the first audio sample in the target scene and the text representation vector of its corresponding text attribute information, as well as the similarity between the audio representation vector of the first audio sample and the text representation vector of other text attribute information.

[0072] Furthermore, by utilizing the audio representation contrast loss and audio-text representation contrast loss in the target scene, a first training loss is calculated. Using this first training loss, the first candidate audio model can be iteratively trained. When a first specified number of iterations is reached or the first training loss converges, a second candidate audio model is obtained. The first specified number of iterations can be set as needed and is not specifically limited here.

[0073] Here, the first training loss may also include a first prediction loss. When calculating the first prediction loss, the time-frequency features of each first audio sample can be reconstructed using the audio representation vector of that first audio sample. Then, the first prediction loss is calculated using the time-frequency features of the first audio sample and the features corresponding to the mask portion in the reconstructed time-frequency features of the first audio sample. For example, the mean square error between the time-frequency features of the first audio sample and the features corresponding to the mask portion in the reconstructed time-frequency features of the first audio sample can be used as the first prediction loss.

[0074] The first training loss can be obtained by weighted summation of the first prediction loss, the audio representation contrast loss, and the audio-text representation contrast loss.

[0075] Finally, step S13 is executed to construct an initial audio perception model using the perception task in the target scene and the second alternative audio model. Here, since the second alternative audio model, as an audio representation vector extraction structure, has general audio representation capabilities and audio representation capabilities for several user-focused target scenes, the initial perception task header used to convert the audio representation vectors into the perception results corresponding to the perception task can be determined based on the perception task in the target scene. Then, this initial perception task header is concatenated after the second alternative audio model to obtain the initial audio perception model.

[0076] It is understandable that different types of perception tasks have different initial perception task headers. For example, if the perception task is a classification task, the initial perception header can be a classification layer, which may include fully connected layers and softmax layers; if the perception task is a regression task, the initial perception header can be a regression layer, which may include fully connected layers; if the perception task is a reconstruction task, the initial perception header can be a decoder, which may include deconvolution layers.

[0077] Subsequently, the second audio sample can be used to perform supervised training on the initial audio perception model to enhance its understanding and alignment with the perception task in the target scene, thus constructing an audio representation of the perception task belonging to the target scene. For example, the second audio sample can be input into the initial audio perception model, and the audio representation vector in the second audio sample can be extracted through the second candidate audio model in the initial audio perception model. Then, the audio representation vector in the second audio sample can be converted into the initial perception result through the initial perception task head.

[0078] Using the initial perception result and the task label carried by the second audio sample, a second training loss is calculated. This second training loss is then used to iteratively train the initial large-scale audio perception model until a second specified number of iterations is reached or the second training loss converges, thus obtaining the target large-scale audio perception model. The second specified number of iterations can be set as needed and is not specifically limited here.

[0079] It is understandable that the iterative training process of the initial audio perception model can be either a joint training of the second alternative audio model and the initial perception task head, or training only the initial perception task head; no specific limitation is made here. When calculating the second training loss, the supervised cross-entropy loss function can be used.

[0080] The audio perception large-scale model training method provided in this embodiment of the invention utilizes first audio samples and text attribute information, applies mask prediction criteria, and combines cross-scene audio representation comparison learning and audio-text representation comparison learning within the same scene. Under the target scene, a first candidate audio large-scale model is trained to obtain a second candidate audio large-scale model. This not only fully absorbs the data knowledge of the first audio samples from various scenes, improving the general audio representation capability of the second candidate audio large-scale model in the target scene and exhibiting good generalization, but also effectively avoids knowledge confusion in different scenes, enhancing the audio perception capability of the perception task in the target scene. Through cross-scene audio representation comparison learning, not only can the coverage and memory capacity of knowledge in the target scene be strengthened, but also the ability to perform mixed processing and analysis between different tasks in the target scene can be improved by combining the first audio samples in the target scene with mixed training. Furthermore, introducing text attribute information and combining it with audio-text representation comparison learning within the same scene can increase the probability of the same content being expressed using different audio in the target scene, such as different animal cries of fear. Transfer learning between semantically similar audio can alleviate the problem of low-resource data scarcity. Moreover, completing the training of the first candidate audio model can enhance the general ability of the second candidate audio model to extract audio representation vectors in the target scene, and reduce the repetitiveness of subsequent training of the initial audio perception model.

[0081] Based on the above embodiments, the step of training the first candidate audio model in the target scene by applying a mask prediction criterion based on the first audio sample and the text attribute information, combined with cross-scene audio representation comparison learning and same-scene audio-text representation comparison learning, to obtain the second candidate audio model includes:

[0082] Obtain third-party audio samples for various scenarios;

[0083] Based on the third audio sample, the mask prediction criterion is applied to train the general audio model to obtain the first candidate audio model.

[0084] Specifically, in this embodiment of the invention, the first alternative audio model can be obtained by unsupervised training of a general audio model. For example, a third audio sample of a general multi-scene can be obtained, including but not limited to industrial machine audio, underwater ocean audio, animal call audio, music audio, etc.

[0085] Using the third audio sample, a mask prediction criterion is applied to train a general audio model, resulting in a first candidate audio model. The general audio model can be structured as a neural network, including but not limited to CNN, RNN, LSTM, and Transformer.

[0086] First, the time-frequency features of the third audio sample can be extracted. Then, these features are masked at a fixed ratio, and the masked features are input into a general audio model. The general audio model uses the audio representation vector of the unmasked portion to predict the audio representation vector of the masked portion, thus obtaining the audio representation vector of the third audio sample. Subsequently, the time-frequency features of the third audio sample are reconstructed using this audio representation vector.

[0087] Subsequently, the second prediction loss is calculated using the time-frequency features of the third audio sample and the features corresponding to the mask portion in the time-frequency features of the reconstructed third audio sample. For example, the mean square error between the time-frequency features of the third audio sample and the features corresponding to the mask portion in the time-frequency features of the reconstructed third audio sample can be used as the second prediction loss.

[0088] The second prediction loss can be expressed as:

[0089] ;

[0090] in, For the second predicted loss, The features corresponding to the mask part in the time-frequency features of the reconstructed third audio sample. The features corresponding to the mask part in the time-frequency features of the third audio sample.

[0091] Subsequently, the second prediction loss is used to calculate the third training loss. For example, the second prediction loss can be directly used as the third training loss. The third training loss can be used to iteratively train the general audio model until the third training loss converges or reaches the third specified number of iterations, thus obtaining the first candidate audio model. The third specified number of iterations can be set as needed and is not specifically limited here.

[0092] Therefore, it can be concluded that, Figure 2 As shown, in the process of determining the target audio perception big model, three-level modeling was implemented, from the general audio big model to the first candidate audio big model, from the first candidate audio big model to the second candidate audio big model, and from the second candidate audio big model to the target audio perception big model.

[0093] In this embodiment of the invention, a general audio model is trained by combining third audio samples from multiple scenarios with a mask prediction criterion. Since the data from different scenarios are complementary in terms of knowledge coverage and task coverage, the training by data fusion of third audio samples from multiple scenarios can fully absorb the data knowledge of general audio data and audio from various scenarios, improve the underlying general representation ability of the first candidate audio model in different scenarios, have good generalization ability, and reduce the difficulty and repetition of subsequent model training.

[0094] Based on the above embodiments, the number of the third audio samples includes multiple samples;

[0095] The construction of an initial audio perception model based on the perception task and the second alternative audio model includes:

[0096] For any third audio sample, the third audio sample is segmented to obtain several audio segments, and based on the second candidate audio large model, the first audio representation vector of each audio segment and the second audio representation vector of the third audio sample are extracted.

[0097] Based on the similarity between each of the first audio representation vectors and the second audio representation vectors, each of the third audio samples is screened to determine the fourth audio sample, and based on the fourth audio sample, the first candidate audio large model is trained to obtain the third candidate audio large model.

[0098] Based on the first audio sample and the text attribute information, a mask prediction criterion is applied, and audio representation comparison learning across scenes and audio-text representation comparison learning within the same scene are combined to train the third alternative audio model in the target scene to obtain a fourth alternative audio model. Based on the perception task and the fourth alternative audio model, the initial audio perception model is constructed.

[0099] Specifically, the number of third audio samples can be multiple. When constructing the initial audio perception model, since the general audio model is trained using an unsupervised training method, the third audio samples may contain noisy interference data, which negatively impacts the training of the general audio model. Therefore, to eliminate the negative impact of noisy interference data on the first candidate audio model, the third audio samples can be evaluated using the second candidate audio model. From the third audio samples, training data that can be adapted to the first candidate audio model can be selected, i.e., the fourth audio samples. This improves the audio representation capability of the initial audio perception model for several target scenarios of user interest.

[0100] When filtering the third audio samples, each third audio sample can be segmented, and each third audio sample can be segmented into one or more audio segments. For any third audio sample... By analyzing any third audio sample By segmenting the audio, we can obtain n audio segments, which are represented as follows: , For any third audio sample The nth audio segment obtained by segmentation.

[0101] Each audio segment and third audio samples The audio segments are input into the second alternative audio model, and the first audio representation vector of each segment can be extracted from the second alternative audio model. and the second audio representation vector of the third audio sample .in, For any third audio sample The first audio representation vector of the nth audio segment obtained by segmentation.

[0102] Subsequently, each first audio representation vector can be calculated. With the second audio representation vector The similarity between them is: Where cosine is the cosine function.

[0103] By utilizing the similarity scores of each third audio sample, a fourth audio sample can be selected from them. For example, a third audio sample whose similarity scores are all greater than or equal to a first specified threshold can be used as the fourth audio sample.

[0104] Then, using the fourth audio sample, the first candidate audio model is trained a second time to obtain the third candidate audio model. Here, the process of training the first candidate audio model a second time is the same as the process of training the general audio model described above, and will not be repeated here.

[0105] Subsequently, using the first audio sample and text attribute information, a mask prediction criterion is applied, combined with cross-scene audio representation comparison learning and same-scene audio-text representation comparison learning, to train the third candidate audio model in the target scene, thus obtaining the fourth candidate audio model. The training process for the third candidate audio model is the same as the first training process for the first candidate audio model described above, and will not be repeated here.

[0106] Finally, the initial audio perception model is obtained by splicing the initial perception task head after the fourth alternative audio model.

[0107] In this embodiment of the invention, by screening the third audio samples to perform secondary training on the first candidate audio large model, and then training the obtained third candidate audio large model, the negative impact of noise interference data in the third audio samples on the first candidate audio large model can be eliminated, thereby improving the audio representation capability of the initial audio perception large model in the target scene.

[0108] Based on the above embodiments, the step of filtering each of the third audio samples based on the similarity between each of the first audio representation vectors and the second audio representation vectors to determine the fourth audio sample includes:

[0109] Determine the proportion of each audio segment in each of the third audio samples whose similarity is greater than or equal to a first specified threshold;

[0110] Based on the proportion corresponding to each of the third audio samples, the third audio samples are filtered to determine the fourth audio sample.

[0111] Specifically, when determining the fourth audio sample, the proportion p of each audio segment in each third audio sample with a similarity greater than or equal to a first specified threshold can be calculated first:

[0112] ;

[0113] Where T1 is the first specified threshold, which can be set as needed, for example, it can be set to 0.85.

[0114] By utilizing the proportion of each third audio sample, the third audio samples can be filtered. For example, third audio samples with a proportion greater than or equal to a second specified threshold can be directly selected as fourth audio samples. The second specified threshold can also be set as needed, for example, it can be set to 0.6.

[0115] In this embodiment of the invention, by determining the proportion of each audio segment in each third audio sample whose similarity is greater than or equal to a first specified threshold, and filtering each third audio sample, interference data in the fourth audio sample can be reduced while ensuring the quantity of the fourth audio sample.

[0116] Based on the above embodiments, the step of filtering each of the third audio samples based on the proportion corresponding to each of the third audio samples to determine the fourth audio sample includes:

[0117] A fifth audio sample is determined based on the proportion of each of the third audio samples that is greater than or equal to a second specified threshold, and a fourth audio sample is determined based on each of the audio segments in the fifth audio sample whose similarity is greater than or equal to the first specified threshold.

[0118] Specifically, when determining the fourth audio sample, the third audio samples with a proportion greater than or equal to the second specified threshold in each third audio sample can be used as the fifth audio sample, and the audio segments in the fifth audio sample can be filtered. For example, each fourth audio sample can be constructed by using each audio segment in each fifth audio sample with a similarity greater than or equal to the first specified threshold.

[0119] In this embodiment of the invention, the quality of the fourth audio sample can be further improved by filtering the audio segments in each fifth audio sample.

[0120] Based on the above embodiments, the lengths of the audio segments in the same third audio sample are the same, and the lengths of the audio segments in different third audio samples are randomly determined within a preset length.

[0121] Specifically, when segmenting each third audio sample, different third audio samples can be segmented using different audio segment lengths. That is, the length of each audio segment in the same third audio sample is the same, and the length of each audio segment in different third audio samples is randomly determined within a preset length. This preset length can be set as needed, for example, it can be set to a time length of 10 seconds.

[0122] In this embodiment of the invention, by segmenting different third audio samples using different audio segment lengths, the adverse effects of audio segment length on the screening of third audio samples can be reduced.

[0123] Based on the above embodiments, the first objective of the audio representation comparison learning includes that the similarity of audio representation vectors in the same scene is higher than that of audio representation vectors in different scenes;

[0124] The second objective of the audio-text representation contrastive learning is that the similarity between the audio representation vector and the corresponding text attribute information text representation vector in the same scene is higher than the similarity between the audio representation vector and the text attribute information text representation vector.

[0125] Specifically, the first objective of audio representation contrastive learning is to achieve a higher similarity between audio representation vectors in the same scene and those in different scenes. This first objective can eliminate the influence of differences in scene audio background on audio representation vectors.

[0126] Audio representation vector similarity in the same scene refers to the similarity between the audio representation vectors of two first audio samples in the same scene, which can be expressed as: ;in, It is a cosine function. Let be the audio representation vector of the first audio sample 1 in scenario A. Let be the audio representation vector of the first audio sample 2 in scenario A.

[0127] Audio representation vector similarity in different scenarios refers to the similarity between the audio representation vectors of two first audio samples in different scenarios, which can be expressed as: ;in, Let 3 be the audio representation vector of the first audio sample 3 in scenario B.

[0128] The second objective of audio-text representation contrastive learning is to ensure that the similarity between the audio representation vector and the corresponding text representation vector of text attribute information in the same scene is higher than the similarity between the audio representation vector and the text representation vector of other text attribute information. By achieving this second objective, the influence of scene-related text attribute information on the audio representation vector can be eliminated.

[0129] The similarity between the audio representation vector and the corresponding text attribute information text representation vector in the same scene can be expressed as: ;in, Let be the text representation vector of the text attribute information of the first audio sample 1 in scenario A.

[0130] The similarity between audio representation vectors and text representation vectors of other text attribute information in the same scene can be expressed as: ;in, Let be the text representation vector of the text attribute information of the first audio sample 2 in scenario A.

[0131] Based on the above embodiments, the step of applying a mask prediction criterion based on the first audio sample and the text attribute information, combined with cross-scene audio representation comparison learning and same-scene audio-text representation comparison learning, to train the first candidate audio large model in the target scene to obtain the second candidate audio large model includes:

[0132] The time-frequency features of the first audio sample are determined, and the time-frequency features are masked to obtain masked features. The masked features are then input into the first candidate audio large model to obtain the audio representation vector output by the first candidate audio large model.

[0133] Based on the audio representation vector, calculate the audio representation contrast loss corresponding to the first target in the target scene, and extract the text representation vector of the text attribute information. Based on the audio representation vector and the text representation vector, calculate the audio-text representation contrast loss corresponding to the second target in the target scene.

[0134] Based on the audio representation contrast loss and the audio text representation contrast loss, a first training loss is calculated, and based on the first training loss, the first candidate audio large model is trained to obtain the second candidate audio large model.

[0135] Specifically, when training the first candidate audio large model, the time-frequency features of the first audio sample, such as SEFE features, can be extracted first, and the time-frequency features of the first audio sample can be partially masked according to the proportion. The partially masked time-frequency features are then input into the first candidate audio large model. The first candidate audio large model predicts the audio representation vector of the masked part using the audio representation vector of the unmasked part, thereby obtaining the audio representation vector of each first audio sample.

[0136] Using the audio representation vectors of each first audio sample, the audio representation contrast loss corresponding to the first target in the target scene is calculated using the following formula:

[0137] ;

[0138] in, Let be the audio representation contrast loss corresponding to the first target in the target scene, and max be the maximum value function.

[0139] Using the audio representation vector and text representation vector of each first audio sample, the audio-text representation contrast loss corresponding to the second target in the target scene is calculated using the following formula:

[0140] ;

[0141] in, The audio-text representation contrast loss is used to represent the second target in the target scene.

[0142] Finally, the first training loss is obtained by combining the audio representation contrast loss and the audio-text representation contrast loss with the prediction loss.

[0143] The first training loss can be expressed as:

[0144] , ;

[0145] in, This is the first training loss. To predict losses, To compare learning loss, To control the weights of the loss function, they are usually adjusted based on experiments, and are typically set to 10.

[0146] Finally, the first candidate audio model is trained using the first training loss, and the second candidate audio model can be obtained after training.

[0147] In summary, the audio perception large-scale model training method provided in this embodiment of the invention, based on a three-level modeling process—from a general audio large-scale model to a first candidate audio large-scale model, from the first candidate audio large-scale model to a second candidate audio large-scale model, and from the second candidate audio large-scale model to the target audio perception large-scale model—introduces a model iterative evaluation mechanism. This mechanism uses the downstream second candidate audio large-scale model to evaluate the performance of the third audio samples from the upstream general audio large-scale model, thereby selecting training data more suitable for the upstream first candidate audio large-scale model and performing secondary training on it. During model training, a text large-scale model is introduced, abstracting text attribute information into text representation vectors. Through cross-domain comparative learning, while aligning audio representation vectors from different scenarios, audio representation vectors and text representation vectors from the same scenario are also aligned, enabling the target audio perception large-scale model to obtain audio representation vectors with stronger generalization capabilities.

[0148] like Figure 3 As shown, an embodiment of the present invention provides an audio processing method, including:

[0149] S21, acquire the audio data to be processed for the perception task in the target scene;

[0150] S22, input the audio data to be processed into the target audio perception model to obtain the target perception result of the perception task output by the target audio perception model;

[0151] The target audio perception big model is trained based on the audio perception big model training method provided in the above embodiments.

[0152] Specifically, the audio processing method provided in this embodiment of the invention is executed by an audio processing device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0153] First, step S21 is executed to obtain the audio data to be processed for the perception task in the target scene. This audio data to be processed refers to the audio data used to perform the perception task and determine the perception result corresponding to the perception task.

[0154] Then, step S22 is executed, in which the audio data to be processed is input into the target audio perception large model trained by the audio perception large model training method provided in the above embodiments, the audio representation vector of the audio data to be processed is extracted by the target audio perception large model, and the perception result corresponding to the perception task is determined by the audio representation vector.

[0155] The audio processing method provided in this embodiment of the invention, by applying a target audio perception large model, can make the extracted audio representation vector of the audio data to be processed more accurate, thereby improving the accuracy and reliability of the perception results corresponding to the perception task.

[0156] like Figure 4 As shown, this embodiment of the invention provides an audio perception large model training device, comprising:

[0157] The sample acquisition module 41 is used to acquire the first audio sample in each scene, the text attribute information of the first audio sample, and the second audio sample with the task label of the perception task in the target scene.

[0158] The first training module 42 is used to train the first candidate audio model in the target scene based on the first audio sample and the text attribute information, by applying the mask prediction criterion and combining cross-scene audio representation comparison learning and audio-text representation comparison learning in the same scene, to obtain the second candidate audio model.

[0159] The second training module 43 is used to construct an initial audio perception model based on the perception task and the second alternative audio model, and to train the initial audio perception model based on the second audio sample to obtain the target audio perception model.

[0160] Specifically, the functions of each module in the audio perception large model training device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above-described audio perception large model training method embodiment, and the achieved effect is also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0161] like Figure 5 As shown, an audio processing device is provided in this embodiment of the invention, comprising:

[0162] Data acquisition module 51 is used to acquire audio data to be processed;

[0163] The audio processing module 52 is used to input the audio data to be processed into the target audio perception model and obtain the processing result output by the target audio perception model.

[0164] The target audio perception big model is trained based on the audio perception big model training method provided in the above embodiments.

[0165] Specifically, the functions of each module in the audio processing device provided in this embodiment correspond one-to-one with the operation flow of each step in the above-mentioned audio processing method embodiment, and the achieved effects are also the same. Please refer to the above embodiments for details, and this will not be repeated in this embodiment.

[0166] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the audio perception large model training method or the audio processing method provided in the above embodiments.

[0167] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the audio perception large model training method or the audio processing method provided in the above embodiments.

[0169] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the audio perception large model training method or audio processing method provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and is not specifically limited herein.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio-aware large model training method, characterized in that, include: Acquire the first audio sample in each scenario, the text attribute information of the first audio sample, and the second audio sample with the task label of the perception task in the target scenario; Based on the first audio sample and the text attribute information, a mask prediction criterion is applied, combined with cross-scene audio representation comparison learning and audio-text representation comparison learning within the same scene. In the target scene, the first candidate audio model is trained to obtain the second candidate audio model. The audio representation comparison learning refers to calculating the audio representation comparison loss by utilizing the similarity of audio representation vectors of different first audio samples within the same scene and the similarity of audio representation vectors of first audio samples in different scenes. The audio-text representation comparison learning refers to calculating the audio-text representation comparison loss by utilizing the similarity between the audio representation vector of the first audio sample within the same scene and the text representation vector of the corresponding text attribute information, as well as the similarity between the audio representation vector of the first audio sample and the text representation vector of other text attribute information. Based on the perception task and the second alternative audio model, an initial audio perception model is constructed. Based on the second audio sample, the initial audio perception model is trained to obtain the target audio perception model.

2. The audio-aware large model training method of claim 1, wherein, Based on the first audio sample and the text attribute information, a mask prediction criterion is applied, combined with cross-scene audio representation comparison learning and same-scene audio-text representation comparison learning, to train the first candidate audio model in the target scene, thereby obtaining the second candidate audio model. This process includes: Obtain third-party audio samples for various scenarios; Based on the third audio sample, the mask prediction criterion is applied to train the general audio model to obtain the first candidate audio model.

3. The method for training a large audio perception model according to claim 2, characterized in that, The number of the third audio samples includes multiple samples; the construction of the initial audio perception model based on the perception task and the second candidate audio model includes: For any third audio sample, the third audio sample is segmented to obtain several audio segments, and based on the second candidate audio large model, the first audio representation vector of each audio segment and the second audio representation vector of the third audio sample are extracted. Based on the similarity between each of the first audio representation vectors and the second audio representation vectors, each of the third audio samples is screened to determine the fourth audio sample, and based on the fourth audio sample, the first candidate audio large model is trained to obtain the third candidate audio large model. Based on the first audio sample and the text attribute information, a mask prediction criterion is applied, and audio representation comparison learning across scenes and audio-text representation comparison learning within the same scene are combined to train the third alternative audio model in the target scene to obtain a fourth alternative audio model. Based on the perception task and the fourth alternative audio model, the initial audio perception model is constructed.

4. The method for training a large audio perception model according to claim 3, characterized in that, The step of filtering each of the third audio samples based on the similarity between each of the first audio representation vectors and the second audio representation vectors to determine the fourth audio sample includes: Determine the proportion of each audio segment in each of the third audio samples whose similarity is greater than or equal to a first specified threshold; Based on the proportion corresponding to each of the third audio samples, the third audio samples are filtered to determine the fourth audio sample.

5. The method for training a large audio perception model according to claim 4, characterized in that, The step of filtering each of the third audio samples based on the proportion corresponding to each of the third audio samples to determine the fourth audio sample includes: A fifth audio sample is determined based on the proportion of each of the third audio samples that is greater than or equal to a second specified threshold, and a fourth audio sample is determined based on each of the audio segments in the fifth audio sample whose similarity is greater than or equal to the first specified threshold.

6. The method for training a large audio perception model according to claim 3, characterized in that, The lengths of the audio segments in the same third audio sample are the same, and the lengths of the audio segments in different third audio samples are randomly determined within a preset length.

7. The method for training a large audio perception model according to any one of claims 1-6, characterized in that, The first objective of the audio representation contrastive learning is to achieve a higher similarity between audio representation vectors in the same scene and those in different scenes. The second objective of the audio-text representation contrastive learning is that the similarity between the audio representation vector and the corresponding text representation vector of the text attribute information in the same scene is higher than the similarity between the audio representation vector and the text representation vector of other text attribute information.

8. The method for training a large audio perception model according to claim 7, characterized in that, Based on the first audio sample and the text attribute information, a mask prediction criterion is applied, combined with cross-scene audio representation comparison learning and same-scene audio-text representation comparison learning, to train the first candidate audio model in the target scene, thereby obtaining the second candidate audio model, including: The time-frequency features of the first audio sample are determined, and the time-frequency features are masked to obtain masked features. The masked features are then input into the first candidate audio large model to obtain the audio representation vector output by the first candidate audio large model. Based on the audio representation vector, calculate the audio representation contrast loss corresponding to the first target in the target scene, and extract the text representation vector of the text attribute information. Based on the audio representation vector and the text representation vector, calculate the audio-text representation contrast loss corresponding to the second target in the target scene. Based on the audio representation contrast loss and the audio text representation contrast loss, a first training loss is calculated, and based on the first training loss, the first candidate audio large model is trained to obtain the second candidate audio large model.

9. The method for training a large audio perception model according to any one of claims 1-6, characterized in that, The construction of an initial audio perception model based on the perception task and the second candidate audio model includes: Based on the aforementioned perception task, an initial perception task header is determined; Based on the initial perception task head and the second alternative audio large model, the initial audio perception large model is constructed.

10. An audio processing method, characterized in that, include: Acquire the audio data to be processed for the perception task in the target scene; The audio data to be processed is input into the target audio perception model to obtain the target perception result of the perception task output by the target audio perception model. The target audio perception large model is trained based on the audio perception large model training method as described in any one of claims 1-9.

11. A training device for a large audio perception model, characterized in that, include: The sample acquisition module is used to acquire the first audio sample in each scenario, the text attribute information of the first audio sample, and the second audio sample with the task label of the perception task in the target scenario. The first training module is used to train a first candidate audio model in the target scene based on the first audio sample and the text attribute information, applying a mask prediction criterion and combining cross-scene audio representation comparison learning and audio-text representation comparison learning within the same scene, to obtain a second candidate audio model. The audio representation comparison learning refers to calculating the audio representation comparison loss by using the similarity of audio representation vectors of different first audio samples in the same scene and the similarity of audio representation vectors of first audio samples in different scenes. The audio-text representation comparison learning refers to calculating the audio-text representation comparison loss by using the similarity of the audio representation vector of the first audio sample in the same scene with the text representation vector of the corresponding text attribute information, and the similarity of the audio representation vector of the first audio sample with the text representation vector of other text attribute information. The second training module is used to construct an initial audio perception model based on the perception task and the second alternative audio model, and to train the initial audio perception model based on the second audio sample to obtain the target audio perception model.

12. An audio processing device, characterized in that, include: The data acquisition module is used to acquire the audio data to be processed; An audio processing module is used to input the audio data to be processed into a target audio perception model and obtain the processing result output by the target audio perception model. The target audio perception large model is trained based on the audio perception large model training method as described in any one of claims 1-9.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the audio perception large model training method as described in any one of claims 1-9, or the audio processing method as described in claim 10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the audio perception large model training method as described in any one of claims 1-9, or the audio processing method as described in claim 10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the audio perception large model training method as described in any one of claims 1-9, or the audio processing method as described in claim 10.

Images