An acoustic model training method and apparatus

Abstract

The application provides an acoustic model training method and device. The acoustic model training method provided by the application comprises: acquiring unannotated acoustic samples; pre-training the acoustic model based on a self-supervised method, the acoustic model being used to predict the category of an input acoustic signal, in the self-supervised method, respectively generating masks for the time domain feature and the frequency domain feature of the acoustic signal based on the time domain feature and the frequency domain feature of the acoustic signal, the mask position and the mask quantity of different samples being different; dividing the levels to which the network structures of the pre-trained acoustic model belong, respectively freezing the layers of different levels, and asynchronously fine-tuning the acoustic model, the parameter freezing time of the layers of different levels being different, and the learning rate of the layers of different levels being different. The acoustic model training method and device provided by the application not only reduce the dependence on large-scale artificial annotation data, but also improve the training efficiency and task performance, and better adapt to task requirements and complex application scenarios.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an acoustic model training method and apparatus. Background Technology

[0002] Currently, AI models used for acoustic analysis primarily employ supervised learning methods. These methods typically require collecting large amounts of task-specific acoustic data, which is then manually labeled by domain experts to create a labeled dataset. During model training, the labeled acoustic data is input into a pre-defined neural network for training. Backpropagation optimizes the model parameters, minimizing the error between the model's predicted output and the true labels, thus enabling the model to identify or classify the corresponding acoustic signals. After training, the model is then embedded and deployed in applications. When improving model performance or adapting to new scenarios, it is usually necessary to collect more labeled data and retrain or incrementally train the model. However, existing technologies have several shortcomings. It is difficult to obtain labeled data for supervised learning; the performance of a well-trained model significantly degrades when encountering new data with a distribution inconsistent with the training data. When learning new data, AI models easily forget previously learned knowledge, limiting their continuous learning ability. Furthermore, whenever a large amount of new data or a new task appears, the model needs to be trained from scratch, which is inefficient and resource-intensive. Summary of the Invention

[0003] In view of this, this application provides an acoustic model training method and apparatus to solve the problems of low training efficiency and high computational resource consumption during model training and fine-tuning.

[0004] Specifically, this application is implemented through the following technical solution:

[0005] The first aspect of this application provides an acoustic model training method, the method comprising:

[0006] Obtain unlabeled acoustic samples;

[0007] The acoustic model is pre-trained based on a self-supervised method. The acoustic model is used to predict the category of the input acoustic signal. In the self-supervised method, masks are generated based on the time-domain and frequency-domain features of the acoustic signal, respectively. The mask positions and number of masks are different for different samples.

[0008] The acoustic model is divided into different levels based on the network structure of each layer after pre-training. Layers of different levels are frozen separately, and the acoustic model is then fine-tuned asynchronously. The parameter freezing time and learning rate of layers of different levels are different.

[0009] A second aspect of this application provides an acoustic model training device, the device comprising a modeling module, a training module, and a fine-tuning module; wherein...

[0010] The modeling module is used to acquire unlabeled acoustic samples;

[0011] The training module is used to pre-train the acoustic model based on a self-supervised method. The acoustic model is used to predict the category of the input acoustic signal. In the self-supervised method, masks are generated based on the time-domain and frequency-domain features of the acoustic signal, respectively. The mask positions and number of masks are different for different samples.

[0012] The fine-tuning module is used to classify the network structure of each layer of the pre-trained acoustic model into different levels, freeze layers of different levels, and perform asynchronous fine-tuning on the acoustic model. The parameter freezing time and learning rate of layers of different levels are different.

[0013] This application provides an acoustic model training method and apparatus. Compared to traditional supervised training methods, this invention directly masks acoustic samples, learning the features and correlations of acoustic samples at a deeper and longer scale before model classification, thereby improving the accuracy of model classification. Furthermore, during the fine-tuning stage, hierarchical fine-tuning is used to adapt the model to different tasks, enabling it to adapt to more sound recognition scenarios and improving its generalization ability. Specifically, unlabeled acoustic samples are acquired, and by combining self-supervised pre-training and hierarchical asynchronous fine-tuning, general acoustic features are effectively learned on unlabeled acoustic data. Simultaneously, differentiated freezing strategies and learning rate settings are adopted for different network structures, which not only improves the training efficiency and generalization ability of the acoustic model but also significantly reduces the dependence on large-scale manually labeled data, enabling the acoustic model to quickly adapt to diverse tasks and complex application scenarios. Attached Figure Description

[0014] Figure 1 A flowchart of an embodiment of the acoustic model training method provided in this application;

[0015] Figure 2 A flowchart of Embodiment 2 of the acoustic model training method provided in this application;

[0016] Figure 3 This is a schematic diagram illustrating an acoustic artificial intelligence model training method as an exemplary embodiment of this application;

[0017] Figure 4 This is a schematic diagram of the structure of an embodiment of the acoustic model training device provided in this application. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0019] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0021] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0022] Figure 1 This is a flowchart of an embodiment of the acoustic model training method provided in this application. Please refer to... Figure 1 The acoustic model training method provided in this embodiment may include:

[0023] S101. Obtain unlabeled acoustic samples.

[0024] Specifically, unlabeled acoustic samples are acoustic data that have not been manually categorized. A massive amount of unlabeled acoustic samples covering the target acoustic application scenarios are acquired, containing diverse acoustic events and environmental noise, ensuring data representativeness and scenario coverage. After acquiring the unlabeled acoustic samples, they are formatted using audio processing tools, and denoising algorithms are used to remove extreme noise while preserving the original acoustic features.

[0025] In this specific implementation, a diverse database of over 100,000 hours of unlabeled acoustic data was constructed from public data sources (such as Freesound and AudioSet) and fully anonymized clinical recordings provided by partner institutions. This database covers human speech, various cough sounds, breathing sounds, snoring sounds, and complex real-world environmental noise. The data in this diverse database is stored in the raw waveform representation of the acoustic signals.

[0026] S102. The acoustic model is pre-trained based on a self-supervised method. The acoustic model is used to predict the category of the input acoustic signal. In the self-supervised method, masks are generated based on the time-domain and frequency-domain features of the acoustic signal, respectively. The mask positions and number of masks are different for different samples.

[0027] Specifically, self-supervised methods are machine learning methods that can train models without manually labeled data. They train the model by automatically generating supervisory signals from unlabeled data, enabling the model to learn the inherent structure and feature representation of the input data. An acoustic model is used to predict the category of the input sound signal. As an optional embodiment, the category can be a disease type, and the acoustic model predicts the probability of the disease type for user reference. In this embodiment, for example, the corresponding disease types include various respiratory diseases (such as COPD, asthma, and bronchitis), and the health status is classified as common diseases such as "asthma" and "bronchitis." As another optional embodiment, the category can be a health status, such as healthy or unhealthy, reflecting the probability of a person being in various health states.

[0028] Furthermore, time-domain features are features extracted from the time series of acoustic signals, such as the original waveform, signal amplitude, energy changes, short-time energy, and envelope. Frequency-domain features are frequency representations obtained through acoustic signal transformation, such as power spectrum, Mel spectrum, band energy, and formants. Masking is an operation in the time or frequency domain to mask portions of an acoustic signal, used to train acoustic models to predict the masked segments, thereby learning the internal structure and correlation characteristics of the acoustic signal.

[0029] Understandably, converting unlabeled acoustic samples into time-domain and frequency-domain features is done to extract more information. Time-domain features represent the instantaneous state and structural information of the acoustic signal, facilitating the capture of short-term events. Frequency-domain features represent the spectrum and harmonic distribution of the acoustic signal, which is beneficial for representing frequency characteristics. Combining time-domain and frequency-domain features allows for a comprehensive capture of the dynamic changes and spectral structure characteristics of the acoustic signal over time.

[0030] Specifically, based on acoustic prior knowledge, the aforementioned time-domain and frequency-domain features are masked. This includes two masking methods: masking the time-domain segment of the complete acoustic event and masking the frequency-domain segment of the frequency band containing the fundamental frequency. The acoustic model is trained using time-domain and frequency-domain masks, enabling it to predict the masked acoustic time-domain and acoustic frequency-domain signals based on the unmasked context and physical association.

[0031] It should be noted that by introducing masks on the time and frequency domain features, the acoustic model is pre-trained using a self-supervised method. This method can fully learn the features of acoustic signals without manual annotation, which not only reduces the dependence on manual annotation, but also improves the generalization ability and robustness of the acoustic model, effectively reducing training costs and improving overall training efficiency.

[0032] Specifically, in one possible implementation, the acoustic model is pre-trained based on a self-supervised method, including:

[0033] Unlabeled acoustic samples are input into the acoustic model. The feature transformation layer of the acoustic model converts the acoustic samples into time-domain and frequency-domain features. Based on the type of the unlabeled acoustic samples, a mask for the time-domain and frequency-domain features is generated. The prediction layer predicts the complete acoustic time-domain signal based on the context of the time-domain features after a single mask. The prediction layer is also used to predict the complete acoustic frequency-domain signal based on the physical correlation between the frequency domains of the frequency-domain features after a corresponding single mask. The classification layer predicts the category of the unlabeled acoustic samples based on the acoustic time-domain signal and the acoustic frequency-domain signal, calculates the loss function of the acoustic time-domain signal and the acoustic frequency-domain signal, and updates and trains the acoustic model.

[0034] Specifically, by converting unlabeled acoustic samples into time-domain and frequency-domain features, the acoustic model can simultaneously acquire the time and frequency information of the acoustic signal, enabling the pre-trained features to have a more comprehensive and robust generalization ability.

[0035] Specifically, to train the acoustic model, the acoustic signal is sampled, quantized, and segmented into frames, and temporal features are extracted to determine the temporal segments containing complete acoustic events, ensuring smooth subsequent training. Furthermore, these temporal segments containing complete acoustic events are masked to generate the temporal target signal used for acoustic model training.

[0036] Specifically, masks are generated based on the time-domain and frequency-domain features of the acoustic signal, respectively, including:

[0037] (1) The acoustic signal is sampled and quantized to obtain a discrete time-domain signal. The discrete time-domain signal is framed to obtain several time-domain segments. The time-domain features of the time-domain segments are calculated to obtain the time-domain features of the acoustic signal.

[0038] Specifically, sampling and quantization processes involve sampling the continuous acoustic signal at equal intervals along the time axis and discretizing the amplitude value of each sampling point into a value with finite precision. Frame segmentation divides the long-duration discrete-time signal into short-duration time-domain segments, making the signal approximately stationary within each time-domain segment, which facilitates subsequent feature extraction.

[0039] Understandably, extracting time-domain features can effectively capture the dynamic changes of acoustic signals in the time dimension, ensuring data processability while highlighting features related to the target acoustic event, thus providing more suitable input features for subsequent model training.

[0040] (2) Locate the time-domain segment containing the complete acoustic event in the time-domain features.

[0041] Specifically, a complete acoustic event is an independent acoustic unit that appears completely from beginning to end in an acoustic signal. Examples include a cough sound or the inhalation or exhalation process of a breath sound. Acoustic events exhibit changes in their temporal characteristics, which can be used to determine the boundaries of the acoustic event.

[0042] Understandably, by locating acoustic events, it is possible to ensure that the extracted temporal segments have complete semantics, avoid feature loss due to truncation, and thus improve the recognition accuracy and robustness of the acoustic model during the training process.

[0043] (3) In the time domain segment, based on the preset masking strategy, the time domain segment containing the complete acoustic event is selected for masking to obtain the time domain target signal.

[0044] Specifically, the masking strategy is a set of rules based on requirements, such as the masking ratio (e.g., 15%~30%), mask length, and randomization of mask position. The temporal target signal is the signal obtained after the masking operation, in which some frames or segments are masked.

[0045] In a specific implementation, one possible approach is to use simple speech activity detection (VAD) or energy detection algorithms to automatically identify and mask out the complete acoustic event, forcing the acoustic model to learn to reconstruct the complete acoustic signal based on the contextual acoustic information.

[0046] Understandably, by masking the temporal features of acoustic signals, acoustic models are forced to recover the masked temporal segments based on contextual information, thereby learning the temporal structure of acoustic signals and improving the representational ability and adaptability to unlabeled data of self-supervised pre-trained acoustic models.

[0047] As an optional embodiment, an acoustic event reference sample is determined. Based on the acoustic event reference sample, a time-domain segment containing the complete acoustic event is located in the time-domain features. The located time-domain segment is at least identical to the time-domain features of one acoustic event reference sample. The complete acoustic event can exist within a single time-domain segment or be distributed across adjacent time-domain segments. Preferably, adjacent time-domain segments containing different parts of the same complete acoustic event are located in the time-domain features based on the acoustic event reference sample. These adjacent time-domain segments are then merged, and the merged time-domain segment is masked. The masking strategy provided in this application masks a complete acoustic event, thereby forcing the model to learn the feature information of the sound context. Under the constraint of accurate prediction, it learns more accurate and deeper feature information, thereby improving the accuracy of subsequent category prediction.

[0048] Specifically, after masking the time-domain features based on the time-domain features, it is also necessary to mask the frequency-domain features based on the frequency-domain features.

[0049] Furthermore, a mask is generated based on the time-domain and frequency-domain features of the acoustic signal, respectively, including:

[0050] (1) The acoustic signal is converted to obtain a Mel spectrum, the frequency domain spectrum is divided into frames to obtain several frequency domain segments, and the frequency domain features of the frequency domain segments are extracted.

[0051] Specifically, the Short-Time Fourier Transform (SFT) divides a long-duration acoustic signal into short time frames and performs a Fourier transform on each frame to obtain a frequency domain spectrum that simultaneously reflects time and frequency information. The SFT is used to map a continuous-time acoustic signal onto another plane, thus reflecting the signal's changes in both time and frequency dimensions.

[0052] (2) Perform fundamental frequency detection on the frequency domain segment to determine the frequency band where the fundamental frequency of the frequency domain segment is located.

[0053] Specifically, the fundamental frequency is the most basic frequency of a periodic acoustic event, corresponding to the pitch of a sound. Fundamental frequency detection is achieved by analyzing the frequency domain characteristics of the signal, which can be done using the spectral peak method or the harmonic multiplication spectrum method. Fundamental frequency processing helps the model learn the intrinsic physical relationships of sound pitch, timbre, and harmonic structure, thereby improving the accuracy of the frequency domain feature representation.

[0054] Furthermore, to accurately identify the fundamental frequency, peak detection can be performed on all frequency components in the spectrogram to obtain a set of spectral peaks for each frequency. Then, the frequency of each frequency's spectral peak appearing in different time frames is counted, and the frequency with the highest frequency is determined as the fundamental frequency of the periodic acoustic event, thus identifying the frequency band in which the fundamental frequency is located. Through this processing, the acoustic model can capture the inherent physical relationships of sound, such as pitch, timbre, and harmonic structure, improving the accuracy and robustness of representing frequency domain characteristics.

[0055] (3) Mask the base frequency and synchronously mask the integer multiples of the base frequency according to the harmonic relationship to obtain the frequency domain target signal.

[0056] Specifically, harmonics refer to the set of frequencies consisting of the fundamental frequency and its integer multiples, where each integer multiple of the fundamental frequency is an integer multiple of the fundamental frequency. Since the fundamental frequency typically contains the main timbre information, its harmonic bands need to be masked simultaneously when masking the fundamental frequency. This process forces the acoustic model to learn the intrinsic physical relationships of pitch, timbre, etc., improving the representational ability and generalization performance of the pre-trained acoustic model using self-supervised methods. The frequency domain target signal is the target signal obtained after masking.

[0057] Furthermore, based on the inherent physical relationship between the fundamental frequency and harmonics in the acoustic signal, the harmonic frequency band range that needs to be synchronized is determined, that is, the frequency values ​​of integer multiples of the fundamental frequency, such as 2, 3, 4, etc., are calculated, and the frequency band area is delineated in the spectrum for masking.

[0058] In practical implementation, for example, the Audio Spectrogram Transformer (AST) based on the Transformer architecture is chosen as the backbone network. Instead of random masking, a structured masking strategy based on acoustic priors is implemented. This involves converting the input audio signal into a Mel spectrogram, identifying segments with complete acoustic events (e.g., coughing) within the Mel spectrogram, and then masking these segments during training. By training the AST model, the acoustic model can predict and reconstruct the masked acoustic content based on the unmasked contextual information, thereby extracting the inherent features and patterns of the acoustic events.

[0059] Understandably, by masking the fundamental frequency and harmonic bands of a frequency domain signal, an acoustic model can learn the frequency structure, pitch, and harmonic relationships of the acoustic signal, thereby improving the generalization ability of self-supervised pre-training and the accuracy of identifying different acoustic events.

[0060] Specifically, time-domain prediction focuses on waveform recovery, while frequency-domain prediction focuses on pitch, harmonics, and timbre characteristics. Contextual information is used to predict occluded regions, and the acoustic model learns the temporal changes and frequency structure of acoustic events, improving the completeness and generalization ability of feature representation.

[0061] Furthermore, the basic architecture of the acoustic model is divided into a feature transformation layer, a prediction layer, and a classification layer. The feature transformation layer converts the input acoustic samples into time-domain and frequency-domain features for subsequent masking and feature processing. Based on the type of the unlabeled acoustic sample, the feature transformation layer generates masks for the time-domain and frequency-domain features. Based on the time-domain features after a single mask, the prediction layer predicts the complete acoustic time-domain signal, learning the temporal changes and waveform characteristics of acoustic events. Based on the frequency-domain features after a single mask, the prediction layer learns the physical relationships between various frequency-domain components, including fundamental frequency, harmonics, and timbre features, thereby predicting the complete acoustic frequency-domain signal. The classification layer combines the predicted acoustic time-domain and frequency-domain signals to predict the category of the unlabeled acoustic sample.

[0062] Furthermore, when constructing positive sample pairs, not only are different versions of the same audio data used, but recordings of the same core acoustic event under different noise backgrounds are also considered as positive sample pairs. This two-dimensional approach to constructing positive sample pairs allows the pre-trained model to be fully exposed to similar acoustic events under various noise scenarios during the initial training phase, giving it a stronger ability to resist environmental noise.

[0063] Specifically, the loss function is based on the difference between the prediction and the unmasked signal, and is used to guide the update of the acoustic model parameters. The pre-trained weights are the acoustic model parameters after self-supervised training; they are the outputs of the self-supervised training of the acoustic model and can be used as initial weights for subsequent asynchronous fine-tuning.

[0064] Furthermore, the loss function includes a self-supervised loss, which consists of a time-domain prediction loss and a frequency-domain prediction loss. The self-supervised loss is used to calculate the predicted complete acoustic time-domain signal and its features, and also to calculate the difference between the predicted complete acoustic frequency-domain signal and its features. Preferably, the model also includes a reward / penalty function, which calculates a reward or penalty based on the matching between the predicted category and the preset target.

[0065] Specifically, the parameters of the acoustic model are trained based on loss and reward / penalty functions to achieve joint learning of time-domain and frequency-domain features. The acoustic model continuously optimizes its predictive ability for time-domain and frequency-domain signals, enabling the pre-trained weights to capture the complete structure and physical laws of acoustic signals, providing high-quality feature representations for downstream tasks.

[0066] In one possible implementation, a prediction layer has two output channels: a time-domain prediction channel and a frequency-domain prediction channel. The time-domain channel calculates the difference between the predicted complete acoustic time-domain signal and the time-domain features. A commonly used metric is the mean square error, as shown in the following formula:

[0067] ;

[0068] in, and Let N represent the predicted complete acoustic time-domain signal and time-domain features of the i-th sample, respectively; N is the number of samples.

[0069] Furthermore, the difference between the predicted complete acoustic frequency domain signal and the frequency domain features is calculated. A commonly used metric is the mean square error, as shown in the following formula:

[0070] ;

[0071] in, and Let M represent the predicted complete acoustic frequency domain signal and frequency domain features of the i-th sample, respectively; M is the number of samples.

[0072] In another possible implementation, a reward / penalty function is used to drive the model closer to a preset target, as shown in the following formula:

[0073]

[0074] Where R is the reward value; The regularization coefficient is used. For parameter regularization terms.

[0075] Furthermore, by weighted summing the time-domain prediction loss, frequency-domain prediction loss, and reward / penalty function, we obtain the total loss function:

[0076]

[0077] in, , and These are hyperparameters used to adjust the weights of the various loss components.

[0078] It should be noted that the acoustic time-domain and frequency-domain signals predicted by the acoustic model are backfilled into the corresponding masked regions to form a complete reconstructed segment. During this process, the predicted segment and the original unmasked features are smoothed at the beginning and end to ensure the continuity and naturalness of the time-domain waveform and frequency-domain spectrum. Preferably, the abrupt change rate of the acoustic features at the beginning and end is calculated. These acoustic features include both time-domain and frequency-domain features. Smoothing is performed based on the relationship between the abrupt change rate and a preset threshold; if the abrupt change rate exceeds the preset threshold, smoothing is performed. The reconstructed segment can be used for self-supervised training evaluation or to provide complete input for downstream tasks.

[0079] Specifically, through self-supervised pre-training based on unlabeled acoustic samples, acoustic models can actively extract the inherent patterns of acoustic events from massive, inexpensive, and even noisy unlabeled acoustic data, thereby significantly reducing the dependence on manually labeled data and realizing the transformation from passively acquiring data to actively learning knowledge, thus significantly improving the performance and generalization ability of the model.

[0080] Understandably, because the pre-trained model already has a deep understanding of the intrinsic structure of acoustics, the fine-tuning process becomes an optimization based on knowledge transfer, enabling the model to have higher accuracy and better learning ability on core tasks. It exhibits excellent robustness and generalization ability when facing new devices, new environments, or unknown noise interference, realizing an intelligent ecosystem that can be deployed once and evolve throughout its life.

[0081] S103. Divide the network structure of each layer of the pre-trained acoustic model into different levels, freeze the layers of different levels, and perform guided asynchronous fine-tuning of the acoustic model. The parameter freezing time and learning rate of the layers of different levels are different.

[0082] Specifically, the network structure consists of different levels of network layers, divided into first-level, second-level, and third-level layers. The first-level layer corresponds to the bottom layer network and is used to learn general spectral primitives. The second-level layer corresponds to the middle layer network and is used to learn acoustic structure combinations. The third-level layer corresponds to the high-level network and task head, and is used to learn task-related semantics. Asynchronous fine-tuning involves using differentiated learning rates and parameter freeze times at different times during the fine-tuning process, rather than training all layers uniformly. This allows the acoustic model to adapt to downstream tasks while maintaining general features. The learning rate is the step size for each parameter update in the optimization algorithm. The parameter freeze time is the duration or number of training epochs during which the network parameters of a certain layer remain unchanged.

[0083] It should be noted that, in one possible implementation, the bottom-layer network consists of the first 2-4 layers of convolutional or self-attention blocks, used to extract general spectral primitives and local acoustic features. During the fine-tuning phase, a first learning rate (e.g., 1e-5) or complete freezing is used to preserve the general acoustic knowledge formed during pre-training to the greatest extent possible. The middle-layer network generally consists of several intermediate layers (e.g., layers 5-8), responsible for learning acoustic structure combinations and time-frequency patterns. During the fine-tuning phase, a second learning rate (e.g., 1e-4) is used, allowing for limited parameter updates to adequately adapt to downstream task requirements. The high-layer network and task head include the last few layers (e.g., layers 9-12) and classification or prediction layers, primarily learning task-related semantic features. During the fine-tuning phase, a third learning rate (e.g., 1e-4) is used for sufficient training. Through this layered unfreezing and differential learning rate strategy, different layers achieve asynchronous updates during the fine-tuning process.

[0084] Specifically, the parameter freezing time and learning rate are different for different levels of layers. The learning rate of the first level layer is less than that of the second level layer, and the learning rate of the second level layer is less than that of the third level layer. The number of frozen parameters in the first level layer is greater than that in the second level layer, and the number of frozen parameters in the second level layer is greater than that in the third level layer.

[0085] Furthermore, in one possible implementation, the hierarchy of each layer of the pre-trained acoustic model's network structure is determined, including:

[0086] Step 1: Determine the task type and the influence of the parameter set of each layer on the classification result in the pre-trained acoustic model. Determine the level of each layer based on the task type. The task type is a sub-task in which the acoustic model performs the classification function. There are multiple levels, and each level corresponds to a learning rate.

[0087] Specifically, the feature transformation layer of the acoustic model is set as the first-level layer. The first-level layer is used to extract locally common spectral primitive features. The acoustic signal is sampled and quantized, and a spectral representation is obtained through short-time Fourier transform. The spectrum is input into a low-level convolutional network or the initial feature extraction layer. The convolutional kernel scans the spectrogram to extract local frequency patterns and energy features, i.e., spectral primitives. The parameters of this layer can be frozen, or a very small first learning rate can be set to minimize updates and preserve pre-trained knowledge.

[0088] Understandably, the prediction layer of the acoustic model is set as a second-level layer. Based on the learning task of learning acoustic structure combinations, this second-level layer, with a second learning rate, is used for appropriate adjustments to adapt to the new task. Acoustic structure combinations are higher-level acoustic event patterns formed by combining low-level spectral primitives. The low-level spectral primitives extracted from the first-level layer are used as input to construct a mid-level feature sequence. Mid-level convolutional or recurrent / attention networks are used to capture temporal and frequency pattern combinations. A second learning rate is set to allow the acoustic model to be fine-tuned appropriately to adapt to downstream tasks. During training, the network progressively optimizes its parameters through backpropagation to learn acoustic structure combinations.

[0089] Specifically, the classification layer of the acoustic model is set as the third-level layer. Based on the learning task related to the semantics of the learning task, the third-level layer is used with a third learning rate for sufficient training to fit the specific task. Task-related semantics are acoustic feature representations closely related to specific downstream tasks and are used to complete classification or recognition. For example, in a respiratory disease detection task, cough, wheezing, or breathing sound patterns in specific frequency bands correspond to the semantic labels of "healthy" or "ill."

[0090] It is understandable that by dividing the acoustic model into hierarchical levels and adopting a hierarchical freezing and learning rate strategy, the transfer efficiency of the acoustic model can be improved and the task adaptability enhanced, while effectively avoiding the loss of low-level knowledge or overfitting, thereby improving the overall classification or recognition performance.

[0091] Step 2: Determine the freezing parameters based on the impact of the level and classification results. The number of freezing parameters varies for different levels.

[0092] Specifically, the number of frozen parameters refers to whether certain layers remain fixed during fine-tuning and do not participate in gradient updates. The parameter freezing process differs for different levels of layers, and the learning rate also differs for different levels of layers. The higher the level, the lower the learning rate. Each level corresponds to a certain number of frozen parameters, with higher levels having more frozen parameters.

[0093] Specifically, the learning rate of the first level layer is less than that of the second level layer, and the learning rate of the second level layer is less than that of the third level layer; the number of frozen parameters in the first level layer is greater than that in the second level layer, and the number of frozen parameters in the second level layer is greater than that in the third level layer.

[0094] In practice, for example, most parameters of the first-level layer are frozen, and training is performed only on the third-level layer, which is strongly correlated with high-level semantics, using a differential learning rate. The initial model fine-tuning process is efficient and stable, typically converging within hours, resulting in an initial task model capable of accurately distinguishing multiple respiratory diseases.

[0095] Understandably, by adopting differentiated learning rates and the number of frozen parameters for different levels, layered adaptation can be achieved during the fine-tuning stage. This not only improves the efficiency of parameter optimization and training stability, but also enables the model to retain existing knowledge and learn new features when adapting to new tasks or different scenarios, thereby significantly improving the generalization ability of the acoustic model and the performance of downstream tasks.

[0096] Specifically, after classifying the network structure of each layer of the pre-trained acoustic model into its respective level and freezing the layers of different levels, the acoustic model needs to be asynchronously fine-tuned.

[0097] Specifically, after asynchronous fine-tuning the acoustic model, the following steps are taken:

[0098] (1) Create multiple training processes, each training process corresponds to a level, multiple layers within a level are trained synchronously, and different training processes are trained asynchronously.

[0099] Specifically, the training process, in the fine-tuning phase, involves an independent execution unit performing a complete forward and backward propagation calculation on the acoustic model. Fine-tuning parameters are a set of parameters used to control the training behavior of each layer during the fine-tuning phase.

[0100] Multiple training processes are created based on the hierarchical structure of the acoustic model. Each training process corresponds to a level and independently manages all layers within that level. Within the same level, multiple layers are fine-tuned in parallel and synchronously to ensure consistency and efficiency in feature learning. For the lower layers, the training process typically uses an extremely low learning rate or freezes the model entirely to maintain the stability of general acoustic knowledge. For the middle layers, the training process allows limited parameter updates at a low learning rate, enabling the model to adapt appropriately to new tasks. For the higher layers and task heads, the training process uses a standard learning rate and is fully unfrozen to achieve sufficient training to capture task-relevant semantic features.

[0101] Understandably, creating an independent training process for each layer ensures that the acoustic model can be independently fine-tuned across different layers or tasks. This maintains the stability of general knowledge at lower layers while allowing for appropriate adjustments at mid-levels and thorough training at higher levels, thereby achieving efficient transfer learning, improving accuracy in downstream tasks, and accelerating training speed and resource utilization.

[0102] (2) For each layer, determine the learning rate and the number of frozen parameters according to the corresponding level.

[0103] Specifically, the implementation principles and processes for determining the learning rate and the amount of frozen parameters based on the corresponding level can be found in the above description, and will not be repeated here.

[0104] Furthermore, pre-trained weights are loaded into the acoustic model, and fine-tuning parameters are assigned to the corresponding level layers. Each training process independently performs forward propagation to calculate the loss. Backpropagation is then performed, updating parameters according to the learning rate of each layer. At the end of each training round, the change in training loss is calculated. If the maximum number of training iterations is reached or the loss value does not decrease significantly, the current training process is stopped. The fine-tuning parameters updated by each process are merged into the acoustic model, completing asynchronous fine-tuning.

[0105] (3) The single variable method is used to calculate the influence of each parameter on the classification result, and the N parameters with the highest influence are determined as frozen parameters, where the value of N is equal to the number of frozen parameters.

[0106] Specifically, the single-variable method involves individually changing each parameter of the neural network to observe its impact on the classification results. A higher impact indicates a more sensitive parameter to the output, while a lower impact indicates a less significant contribution to the task. All parameters are sorted from highest to lowest impact, and the N parameters with the lowest impact are selected. These are then frozen during the fine-tuning phase, meaning they do not participate in gradient updates, to preserve existing knowledge or control training stability.

[0107] Understandably, by employing asynchronous fine-tuning combined with a single-variable method to select frozen parameters, this application achieves fine-grained control over different levels of network layers. Freezing low-impact parameters preserves pre-training knowledge, while updating high-impact parameters adapts to new tasks, thus ensuring model stability and improving fine-tuning efficiency. This avoids catastrophic forgetting caused by full-scale fine-tuning, reduces training resource consumption, and simultaneously improves the adaptability and classification accuracy of downstream tasks. In one possible implementation, the acoustic model can utilize a closed-loop feedback mechanism to enable the entire training platform to possess self-diagnosis and self-improvement capabilities, ensuring that model performance continuously approaches theoretical limits. Specifically, for example, it was found that the acoustic model's classification accuracy was significantly lower than that of samples collected in a clinical setting when processing COPD samples from a home environment with low signal-to-noise ratios. The acoustic model further identified key factors leading to performance degradation, including home environment noise and weak patient vital signs. Based on these diagnostic results, optimization instructions were generated, namely, to mine targeted difficult samples or initiate a new round of reinforcement fine-tuning.

[0108] Furthermore, mining targeted hard samples is to supplement targeted data, including using acoustic fingerprints in unlabeled databases to screen out audio segments with similar acoustic features to known hard samples, or performing data augmentation on existing labeled samples by mixing the original audio with real environmental noise (such as television, air conditioning, conversations, etc.) to simulate low signal-to-noise ratio environments and generate more challenging training samples.

[0109] Specifically, the new round of enhancement and fine-tuning involves adjusting model parameters using knowledge distillation. Knowledge distillation involves using the output or intermediate features of the teacher model as training targets to train the student model, optimizing its predictive performance on difficult samples while maintaining its ability to recognize the original training samples. Furthermore, by identifying target difficult samples, the Focal Loss loss function can be used to force the model to focus more attention on these difficult-to-distinguish samples, thereby completely overcoming the identified performance shortcomings.

[0110] The acoustic model training method provided in this embodiment, compared to traditional supervised training, directly masks acoustic samples, allowing for deeper and longer-term learning of the features and correlations of acoustic samples before model classification, thereby improving the accuracy of model classification. Furthermore, during the fine-tuning stage, hierarchical fine-tuning is used to adapt the model to different tasks, enabling it to adapt to more sound recognition scenarios and improving its generalization ability. Specifically, by acquiring unlabeled acoustic samples and combining self-supervised pre-training with hierarchical asynchronous fine-tuning, general acoustic features are effectively learned from unlabeled acoustic data. Simultaneously, differentiated freezing strategies and learning rate settings are adopted for different network structures, which not only improves the training efficiency and generalization ability of the acoustic model but also significantly reduces the reliance on large-scale manually labeled data, enabling the acoustic model to quickly adapt to diverse tasks and complex application scenarios.

[0111] Figure 2 The flowchart for Embodiment 2 of the acoustic model training method provided in this application is shown below. Please refer to... Figure 2 Based on the above embodiments, the step of determining the adjustment strategy of the acoustic model after deployment based on the differences includes:

[0112] S201. Based on the differences, determine the real-time task type to which the real-time sound to be classified belongs, wherein the real-time task type includes at least domain drift tasks and new tasks.

[0113] Specifically, real-time task types are categorized based on the differences between the real-time sound to be classified and the existing training data, including domain-drift tasks and new tasks. Domain-drift refers to a shift in the feature distribution of the real-time sound to be classified compared to the unlabeled acoustic samples, while the task objective remains unchanged. New tasks refer to tasks whose task category or objective is not entirely the same as that of the unlabeled acoustic samples.

[0114] S202. If the real-time task type is a domain drift task, determine the network layer to be adjusted based on the difference, reduce the value of the penalty function, and retrain the parameters of the network layer to be adjusted.

[0115] Specifically, the edge node receives new data; it is a computing node close to the data source and possesses certain computing, storage, and data processing capabilities. The discriminator analyzes the new input data, determining whether it belongs to domain drift or a new task, and dynamically adjusts the execution strategy of the continuous learning algorithm based on the discriminant results. In this embodiment, for example, an elastic weight consolidation algorithm is used. The adjustment strategy is that when it is determined to be a domain drift task, the elastic weight consolidation algorithm reduces the value of the penalty function, thereby allowing for a larger adjustment of the model parameters to adapt to the new data distribution. Specifically, for example, a continuous learning method based on uncertainty assessment is proposed for domain drift tasks. After the initial model is deployed, when applied to the severe COPD patient population of Hospital A, because the acoustic characteristics of this group (e.g., loudness, energy, duration of cough) differ from those of outpatients in the initial training set, the model, while still outputting COPD, significantly reduces its prediction confidence, or its internally generated feature vectors deviate from the existing COPD feature cluster center, due to differences in the acoustic characteristics of this group (e.g., loudness, energy, duration of cough) compared to those of outpatients in the initial training set.

[0116] It should be noted that, in order to carry out knowledge expansion training, the elastic weight consolidation algorithm is used to retrain using atypical samples to expand the knowledge boundary of the COPD category. Through this training process, the model can gradually expand the feature boundary of the COPD category to cover more diverse acoustic performances.

[0117] Furthermore, after completing the above training, Hospital A's server encrypts the updated model parameter changes and uploads them to the central server. The central server then uses a secure aggregation method (e.g., federated averaging) to fuse this update with model updates from other hospitals (e.g., optimizations for bronchitis identification) to obtain a global model with a deeper understanding of COPD categories and greater robustness to diverse acoustic features. This global model is then distributed to the participating hospitals. This achieves continuous expansion of the model's knowledge boundaries and optimization of global performance while ensuring data security.

[0118] S203. If the real-time task type is a new task, increase the value of the penalty function and retrain the acoustic model based on the adjusted penalty function. The retrained acoustic model's ability to recognize the original training samples remains unchanged.

[0119] When a task is identified as new, the elastic weight consolidation algorithm increases the value of the penalty function on the model to solidify and protect the knowledge boundary of the original task and prevent the performance of old tasks from deteriorating.

[0120] Furthermore, after asynchronously fine-tuning the acoustic model, the method further includes:

[0121] The fine-tuned acoustic model is deployed, real-time sound to be classified is acquired, and a discriminator determines the difference between the real-time sound to be classified and the unlabeled acoustic samples used to train the acoustic model. Based on the difference, an adjustment strategy for the deployed acoustic model is determined, and the deployed acoustic model is adjusted according to the adjustment strategy. The standard acoustic samples are then input into the adjusted acoustic model to predict the category of the real-time sound to be classified. It is understood that the discriminator at the edge can identify new knowledge types and adaptively adjust the learning strategy, enabling the acoustic model to efficiently learn new knowledge while effectively overcoming catastrophic forgetting. Compared to traditional training methods, the strategy of combining fine-tuning and continuous learning reduces iteration costs, shortening the acoustic model update cycle from weeks to hours, and improving the utilization of computing resources. Simultaneously, the federated learning framework ensures that the sensitive original data of each participant remains local, exchanging only encrypted acoustic model parameters, fundamentally guaranteeing data security and privacy.

[0122] Figure 3 This is a schematic diagram illustrating an acoustic artificial intelligence model training method as an exemplary embodiment of this application. Please refer to... Figure 3 The diagram illustrates a dynamic and adaptive closed-loop training platform implementation, comprising a large-scale unlabeled acoustic database (310), a small-scale labeled dataset (320), a pre-trained base model (410), an initial task model (510), a central server (610), and edge terminals (Hospital A) (620) and (Hospital B) (620). The diagram is divided into three phases: acoustic structure-aware self-supervised training (400), guided supervised fine-tuning (500), and task boundary-aware federated continuous learning (600).

[0123] It should be noted that the first stage is self-supervised training of acoustic structure awareness (400), which inputs a large-scale unlabeled acoustic database (310) for structured masking and noise-resistant contrastive learning. After the pre-trained base model (410) is passed, it enters the second stage, namely guided supervised fine-tuning (500), which inputs a small-scale labeled dataset (320) for hierarchical unfreezing and differential learning rate. Then, through the initial task model (510), it enters the third stage, namely federated continuous learning of task boundary awareness (600), where the central server (610) distributes the global model to the edge terminals (Hospital A) (620) and (Hospital B) (620). The edge terminal (Hospital A) (620) updates parameters through a discriminator and dynamic continuous learning, while the edge terminal (Hospital B) (620) updates local data and parameters. At the same time, the central server (610) also provides feedback to the large-scale unlabeled acoustic database (310) and the small-scale labeled dataset (320).

[0124] Figure 4This is a schematic diagram of an embodiment of an acoustic model training device provided in this application. Please refer to... Figure 4 The apparatus provided in this embodiment includes a modeling module 410, a training module 420, and a fine-tuning module 430; wherein,

[0125] The modeling module 410 is used to acquire unlabeled acoustic samples;

[0126] The training module 420 is used to pre-train the acoustic model based on a self-supervised method. The acoustic model is used to predict the category of the input acoustic signal. In the self-supervised method, masks are generated based on the time-domain and frequency-domain features of the acoustic signal, respectively. The mask positions and number of masks are different for different samples.

[0127] The fine-tuning module 430 is used to classify the network structure of each layer of the pre-trained acoustic model into different levels, freeze layers of different levels, and perform asynchronous fine-tuning on the acoustic model. The parameter freezing time and learning rate of layers of different levels are different.

[0128] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An acoustic model training method, characterized in that, The method includes: Obtain unlabeled acoustic samples; The acoustic model is pre-trained based on a self-supervised method. The acoustic model is used to predict the category of the input acoustic signal. The acoustic model is divided into a feature transformation layer, a prediction layer, and a classification layer. The feature transformation layer converts the input acoustic samples into time-domain and frequency-domain features. The feature transformation layer generates masks for temporal and frequency domain features based on the type of the unlabeled acoustic samples; The prediction layer predicts the complete acoustic time-domain signal based on the temporal features after a single mask. The prediction layer learns the physical relationships between frequency domain components based on the frequency domain features after a single mask, and predicts the complete acoustic frequency domain signal. The classification layer combines the predicted acoustic time-domain signal and acoustic frequency-domain signal to predict the category of unlabeled acoustic samples. The prediction layer has two output channels: the time-domain channel calculates the difference between the acoustic time-domain signal and the time-domain features, and the frequency-domain channel calculates the difference between the acoustic frequency-domain signal and the frequency-domain features. The time-domain prediction loss, the frequency-domain prediction loss, and the reward / penalty function are weighted and summed to obtain the total loss function. In the self-supervised method, a mask is generated based on the time-domain and frequency-domain features of the acoustic signal, respectively. This includes: sampling and quantizing the acoustic signal to obtain a discrete time-domain signal; performing frame-segmentation on the discrete time-domain signal to obtain several time-domain segments; and calculating the time-domain features of the time-domain segments to obtain the time-domain features of the acoustic signal. Locate the temporal segment containing the complete acoustic event within the temporal features; In the time domain segment, based on a preset masking strategy, the time domain segment containing the complete acoustic event is selected for masking to obtain the time domain target signal; The acoustic signal is converted into a Mel spectrogram, the Mel spectrogram is segmented into frames to obtain several frequency domain segments, and the frequency domain features of the frequency domain segments are extracted. The fundamental frequency of the frequency domain segment is detected to determine the frequency band in which the fundamental frequency of the frequency domain segment is located; the fundamental frequency is masked, and integer multiples of the fundamental frequency are masked synchronously according to the harmonic relationship to obtain the target signal in the frequency domain. The mask position and number of masks are different for different samples. The acoustic model is divided into different levels based on the network structure of each layer after pre-training. Layers of different levels are frozen, and the acoustic model is then subjected to guided asynchronous fine-tuning. The parameter freezing time and learning rate of layers of different levels are different.

2. The method according to claim 1, characterized in that, The acoustic model is pre-trained based on a self-supervised method, including: Unlabeled acoustic samples are input into the acoustic model. The feature transformation layer of the acoustic model converts the acoustic samples into time-domain and frequency-domain features. Based on the type of the unlabeled acoustic samples, a mask for the time-domain and frequency-domain features is generated. The prediction layer predicts the complete acoustic time-domain signal based on the context of the time-domain features after a single mask. The prediction layer predicts the complete acoustic frequency-domain signal based on the physical correlation between the frequency domains of the frequency-domain features after a corresponding single mask. The classification layer predicts the category of the unlabeled acoustic samples based on the acoustic time-domain signal and the acoustic frequency-domain signal, calculates the loss function of the acoustic time-domain signal and the acoustic frequency-domain signal, and updates and trains the acoustic model.

3. The method according to claim 1, characterized in that, The pre-trained acoustic model is classified into different levels based on the network structure of each layer, including: The influence of the task type and parameter set of each layer in the pre-trained acoustic model on the classification result is determined. Based on the task type, the level to which each layer belongs is determined. The task type is a sub-task of the acoustic model to achieve the classification function. There are multiple levels, and each level corresponds to a learning rate. The freezing parameters are determined based on the impact of the level and classification results, and the number of freezing parameters varies for different levels.

4. The method according to claim 3, characterized in that, The parameter freezing process differs for different layer levels, and the learning rate also differs for different layer levels. The higher the level, the lower the learning rate. Each level corresponds to a certain number of frozen parameters, and the higher the level, the more frozen parameters there are.

5. The method according to claim 1, characterized in that, Asynchronous fine-tuning of the acoustic model includes: Multiple training processes are created, each corresponding to a level. Multiple layers within a level are trained synchronously, while different training processes train asynchronously. For each layer, determine the learning rate and the number of frozen parameters according to the corresponding level; The single-variable method is used to calculate the influence of each parameter on the classification result, and the top N parameters in terms of influence are determined as frozen parameters, where the value of N is equal to the number of frozen parameters.

6. The method according to claim 5, characterized in that, After asynchronously fine-tuning the acoustic model, the method further includes: The fine-tuned acoustic model is deployed to acquire real-time sound to be classified. Based on the discriminator, the difference between the real-time sound to be classified and the unlabeled acoustic samples trained by the acoustic model is determined. Based on the difference, the adjustment strategy of the deployed acoustic model is determined. Based on the adjustment strategy, the deployed acoustic model is adjusted. The unlabeled acoustic samples are input into the adjusted acoustic model to predict the category to which the real-time sound to be classified belongs.

7. The method according to claim 1, characterized in that, After asynchronously fine-tuning the acoustic model, the method further includes: The real-time task type to which the real-time sound to be classified belongs is determined based on the differences, and the real-time task type includes at least domain drift tasks and new tasks; If the real-time task type is a domain drift task, determine the network layer to be adjusted based on the difference, reduce the value of the penalty function, and retrain the parameters of the network layer to be adjusted. If the real-time task type is a new task, the value of the penalty function is increased, and the acoustic model is retrained based on the adjusted penalty function. The retrained acoustic model still retains the same ability to recognize the original training samples.

8. An acoustic model training method apparatus, characterized in that, The device includes a modeling module, a training module, and a fine-tuning module; wherein... The modeling module is used to acquire unlabeled acoustic samples; The training module is used to pre-train the acoustic model based on a self-supervised method. The acoustic model is used to predict the category of the input acoustic signal. The acoustic model is divided into a feature transformation layer, a prediction layer, and a classification layer. The feature transformation layer converts the input acoustic samples into time-domain and frequency-domain features. The feature transformation layer generates masks for temporal and frequency domain features based on the type of the unlabeled acoustic samples; The prediction layer predicts the complete acoustic time-domain signal based on the temporal features after a single mask. The prediction layer learns the physical relationships between frequency domain components based on the frequency domain features after a single mask, and predicts the complete acoustic frequency domain signal. The classification layer combines the predicted acoustic time-domain signal and acoustic frequency-domain signal to predict the category of unlabeled acoustic samples. The prediction layer has two output channels: the time-domain channel calculates the difference between the acoustic time-domain signal and the time-domain features, and the frequency-domain channel calculates the difference between the acoustic frequency-domain signal and the frequency-domain features. The time-domain prediction loss, the frequency-domain prediction loss, and the reward / penalty function are weighted and summed to obtain the total loss function. In the self-supervised method, a mask is generated based on the time-domain and frequency-domain features of the acoustic signal, respectively. This includes: sampling and quantizing the acoustic signal to obtain a discrete time-domain signal; performing frame-segmentation on the discrete time-domain signal to obtain several time-domain segments; and calculating the time-domain features of the time-domain segments to obtain the time-domain features of the acoustic signal. Locate the temporal segment containing the complete acoustic event within the temporal features; In the time domain segment, based on a preset masking strategy, the time domain segment containing the complete acoustic event is selected for masking to obtain the time domain target signal; The acoustic signal is converted into a Mel spectrogram, the Mel spectrogram is segmented into frames to obtain several frequency domain segments, and the frequency domain features of the frequency domain segments are extracted. The fundamental frequency of the frequency domain segment is detected to determine the frequency band in which the fundamental frequency of the frequency domain segment is located; The fundamental frequency is masked, and integer multiples of the fundamental frequency are synchronously masked according to the harmonic relationship to obtain the target signal in the frequency domain. The mask position and number of masks are different for different samples. The fine-tuning module is used to classify the network structure of each layer of the pre-trained acoustic model into different levels, freeze layers of different levels, and perform asynchronous fine-tuning on the acoustic model. The parameter freezing time and learning rate of layers of different levels are different.

Images