Prototype learning-based unbalanced distribution speech classification method and device

Abstract

This application relates to a method and apparatus for classifying imbalanced speech based on prototype learning. The method includes: acquiring original speech samples to construct speech samples; constructing a majority class prototype baseline model and a minority class boundary reinforcement model; jointly training the majority class prototype baseline model and the minority class boundary reinforcement model, wherein the majority class prototype baseline model is trained using only majority class speech samples, establishing stable majority class prototype anchors in the feature space and saving optimal encoder parameters, while the minority class boundary reinforcement model is trained using all speech samples including the minority class; and performing speech classification inference based on the trained model to obtain the classification result. This method achieves refined extraction and category recognition of speech features, effectively solving the problems of unstable classification results and performance degradation caused by extreme class imbalance and inconsistent speech duration, and improving the accuracy and robustness of speech classification in complex environments.

Description

Technical Field

[0001] This application relates to the field of speech processing and classification technology, and in particular to a non-equilibrium speech classification method and apparatus based on prototype learning. Background Technology

[0002] With the rapid development of artificial intelligence technology, feature modeling and automatic classification of speech signals have become the core research direction in the field of intelligent speech processing, and have broad application prospects in scenarios such as intelligent security monitoring, industrial acoustic detection, human-computer interaction, and recognition of specific acoustic events.

[0003] In recent years, deep learning technologies, represented by pre-trained speech models such as Wav2vec 2.0 and HuBERT, have significantly improved the ability to model speech features. Through end-to-end learning, they can extract more discriminative high-dimensional representations from raw audio, overcoming the limitations of traditional handcrafted features. However, in real-world applications, speech data often exhibits typical long-tail distribution characteristics, along with problems such as small sample sizes and extreme class imbalance. When processing such datasets, existing speech classification methods tend to over-bias the majority class, resulting in extremely low sensitivity to the crucial minority class, unstable classification performance, and severe performance degradation, greatly affecting the reliability of speech classification methods for practical application.

[0004] Therefore, there is an urgent need in related technologies for a classification method that can adapt to unbalanced distributed speech datasets, effectively improve the recognition accuracy of the minority class while maintaining the classification performance of the majority class, and enhance the robustness of the model in complex environments. Summary of the Invention

[0005] Therefore, it is necessary to provide a non-equilibrium speech classification method and apparatus based on prototype learning to address the aforementioned technical problems.

[0006] Firstly, this application provides a non-equilibrium speech classification method based on prototype learning. The method includes: Obtain the original speech and construct speech samples; Construct a prototype benchmark model for the majority of categories and a boundary reinforcement model for the minority of categories; The majority class prototype baseline construction model and the minority class boundary reinforcement model are jointly trained. The majority class prototype baseline construction model is trained using only majority class speech samples to establish a stable majority class prototype anchor point in the feature space and save the optimal encoder parameters. The minority class boundary reinforcement model is trained using the full set of speech samples containing minority classes to define the minority class decision boundary while maintaining the stability of the majority class distribution. Based on the trained model, speech classification inference is performed to obtain the classification result.

[0007] Optionally, in one embodiment of this application, the majority-class prototype benchmark construction model includes a distillation feature characterizer, a classification mapper, and a majority-class prototype anchor builder, wherein: The distillation feature representation extracts frame-level sequence representations after preprocessing the original speech, and generates fixed-dimensional speech feature embeddings through learnable vector groups and cross-attention mechanisms. The classification mapper performs non-linear remapping of the speech feature embeddings through multiple layers of linear blocks; The majority class prototype anchor builder uses an exponential moving average algorithm to dynamically update the central representation of the majority class, and combines it with scaled cosine similarity to calculate the loss to establish the majority class prototype anchor.

[0008] Optionally, in one embodiment of this application, the distillation feature characterizer preprocesses the original speech by including: To align the time dimension, zero-value padding is performed on the tail of the original speech frames with inconsistent lengths within the batch, and a Boolean mask is generated to mark the valid speech frames and the padding frames.

[0009] Optionally, in one embodiment of this application, the joint loss used in the majority category prototype benchmark construction model consists of classification loss, prototype cohesion loss, and prototype separation regularization loss. The model parameters are jointly updated through backpropagation, so that similar features are aggregated and dissimilar prototypes are separated.

[0010] Optionally, in one embodiment of this application, the minority class boundary reinforcement model includes a distillation feature characterizer and a minority class prototype anchor builder, wherein: The distillation feature characterizer loads and freezes the encoder parameters of the majority-class prototype benchmark model, and extracts minority-class discriminative feature embeddings through a new set of learnable vectors. The minority class prototype anchor builder uses an exponential moving average algorithm to construct minority class prototype centers and combines them with majority class prototype anchors to calculate the similarity between samples and prototypes of all classes.

[0011] Optionally, in one embodiment of this application, the minority class prototype anchor builder extracts the maximum similarity between the test speech and all majority class prototype anchors using the maximum aggregation operator, which is used as the majority class aggregation score. At the same time, it calculates the similarity between the test speech and the minority class prototype center as the minority class prototype score, and completes the determination of minority class and non-minority class based on the two scores.

[0012] Optionally, in one embodiment of this application, the joint loss used by the minority class boundary strengthening model consists of binary cross-entropy loss and noise contrast estimation loss. It strengthens the discriminative boundary between the minority class and the majority class by updating the minority class-specific learnable vector and the minority class prototype center parameter through backpropagation.

[0013] Optionally, in one embodiment of this application, the joint training of the majority-class prototype benchmark construction model and the minority-class boundary reinforcement model includes: Select majority class speech samples to train the majority class prototype benchmark to build the model, and save the optimal model parameters and the convergent and stable majority class prototype anchor point; Initialize the minority class boundary reinforcement model, load and freeze the optimal model parameters and the majority class prototype anchor point; A minority class boundary enhancement model is trained using a full set of speech samples containing minority class data. The original speech labels are remapped to binary labels for both minority and non-minority classes, and only the minority class-specific learnable vector and minority class prototype center parameters are updated.

[0014] Optionally, in one embodiment of this application, the step of performing speech classification inference based on the trained model to obtain the classification result includes: Input the speech to be tested into the minority class boundary enhancement model and calculate the scaling cosine similarity between the speech features and the minority class prototype center and the majority class prototype anchor point. If the similarity to the minority class prototype is dominant, the speech to be tested is determined to be a minority class and the result is output. Otherwise, the test speech is input into the majority class prototype baseline to build a model, determine the specific category of the test speech in the majority class, and output the result.

[0015] Secondly, this application also provides a non-equilibrium speech classification device based on prototype learning. The device includes: The voice data acquisition module is used to acquire raw voice data and construct voice samples. The speech classification model building module is used to build a prototype baseline model for the majority class and a boundary reinforcement model for the minority class. The speech classification model training module is used to perform joint training on the majority class prototype baseline construction model and the minority class boundary reinforcement model. The majority class prototype baseline construction model is trained using only majority class speech samples, establishes a stable majority class prototype anchor point in the feature space and saves the optimal encoder parameters, and the minority class boundary reinforcement model is trained using the full set of speech samples containing minority classes, and delineates the minority class decision boundary while maintaining the stability of the majority class distribution. The speech classification module is used to perform speech classification inference based on the trained model to obtain the classification result.

[0016] The aforementioned unbalanced speech classification method and apparatus based on prototype learning constructs speech samples by acquiring original speech; builds a majority class prototype baseline model and a minority class boundary reinforcement model; performs joint training on the majority class prototype baseline model and the minority class boundary reinforcement model, wherein the majority class prototype baseline model is trained using only majority class speech samples, establishing stable majority class prototype anchors in the feature space and saving optimal encoder parameters, and the minority class boundary reinforcement model is trained using all speech samples including the minority class, defining the minority class decision boundary while maintaining the stability of the majority class distribution; and performs speech classification inference based on the trained model to obtain the classification result. Compared with existing technologies, this invention achieves refined extraction and category recognition of speech features, effectively solving the problems of unstable classification results and performance degradation caused by extreme class imbalance and inconsistent speech duration, and improving the accuracy and robustness of speech classification in complex environments. Attached Figure Description

[0017] Figure 1 This is a diagram illustrating the application environment of an unbalanced speech classification method based on prototype learning in one embodiment. Figure 2 This is a flowchart illustrating an unbalanced speech classification method based on prototype learning in one embodiment. Figure 3 This is a schematic diagram of the structure of a prototype benchmark construction model for multiple categories in one embodiment; Figure 4 This is a schematic diagram of the structure of a distillation feature characterizer in one embodiment; Figure 5 This is a schematic diagram of the structure of a minority category boundary reinforcement model in one embodiment; Figure 6 This is a block diagram of an unbalanced speech classification device based on prototype learning in one embodiment. Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] The unbalanced speech classification method based on prototype learning provided in this application can be applied to, for example... Figure 1The application environment is illustrated. The terminal communicates with the server via a network. The data storage system stores the data the server needs to process. This system can be integrated onto the server, or it can be hosted in the cloud or on other network servers. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0020] In one embodiment, such as Figure 2 As shown, a non-equilibrium speech classification method based on prototype learning is presented, which can be applied to... Figure 1 Taking the server in the example, the following steps are included: S201: Obtain the original speech and construct speech samples.

[0021] In this embodiment, the original audio data for each category to be classified is first collected. The original audio data is then manually reviewed and filtered using algorithms to remove invalid samples that cannot extract effective acoustic features due to strong environmental noise, signal overload (clipping), or insufficient volume. Valid samples are then finely labeled to establish a mapping relationship between sample features and labels. Based on the number of samples in each category, the category with the fewest samples in the dataset is classified as the minority class, and the rest as the majority class. To suppress the performance degradation of minority class recognition caused by majority class samples dominating the training process, a balanced sampling strategy is adopted for the test set, while adhering to an 8:2 training set to test set ratio. This ensures that the number of samples in each category is completely consistent during the testing phase, thereby achieving an unbiased evaluation of the model's discriminative ability.

[0022] S202: Construct a prototype benchmark model for the majority of classes and a boundary reinforcement model for the minority of classes.

[0023] In one embodiment of this application, the majority-class prototype benchmark construction model includes a distillation feature characterizer, a classification mapper, and a majority-class prototype anchor builder, wherein: S301: The distillation feature representation extracts frame-level sequence representations after preprocessing the original speech, and generates fixed-dimensional speech feature embeddings through learnable vector groups and cross-attention mechanisms.

[0024] S302: The classification mapper performs nonlinear remapping of the speech feature embedding through multiple layers of linear blocks.

[0025] S303: The majority class prototype anchor builder uses the exponential moving average algorithm to dynamically update the central representation of the majority class, and combines the scaling cosine similarity to calculate the loss to establish the majority class prototype anchor.

[0026] In one embodiment of this application, a prototype benchmark construction model for multiple categories is constructed, such as Figure 3 As shown, it includes a distillation feature representation unit, a classification mapper, and a multi-class prototype anchor builder. Specifically, the raw audio is first input into the model's distillation feature representation unit. At this point, the pre-trained model Wav2vec 2.0 in the distillation feature representation unit is unfrozen and trained together with a set of learnable vectors, outputting a fixed-dimensional speech feature vector. .

[0027] In a classification mapper, the feature vector Upon entering the network, it sequentially passes through four non-linear feature extraction blocks. Each block strictly adheres to a three-stage processing paradigm: first, a fully connected layer performs linear combination and spatial transformation on the input features; then, a group normalization mechanism is introduced, dividing the feature channels into a fixed number of subgroups for independent standardization to effectively eliminate internal covariate bias and improve the model's convergence stability under mini-batch training conditions; finally, an activation function is applied, endowing the network with sparse response characteristics and non-linear modeling capabilities. This structural design aims to perform non-linear remapping on highly abstract speech embeddings to capture deep acoustic details. Subsequently, the mapping layer performs a final linear projection on the processed features, thereby generating class-logic values. :

[0028] In the formula, This represents the end-to-end mapping function mentioned above. This represents the number of speech categories.

[0029] Most category prototype anchor builders maintain a prototype matrix. To construct global representations for each category, and to dynamically update them using the exponential moving average (EMA) mechanism. During training, for the [specific category]... Each batch of data, defined To be marked as a category The sample index set, whose sample size is expressed as The prototype center for this category performs a weighted fusion based on the feature mean of the current batch and the historical state. The specific update formula is as follows:

[0030] After completing the prototype update, the prototype matrix will be... Normalization Subsequently, the input feature embedding is calculated. The scaled cosine similarity between the normalized prototype matrix and the normalized prototype matrix generates the class probability values ​​for the majority class prototype anchor builder. This process utilizes a scaling factor. To enhance the distinction between categories, its mathematical expression is:

[0031] It should be noted that, Figure 3 This example illustrates the model construction and prototype learning process using only three majority classes, but this is merely illustrative and does not constitute a limitation on the scope of this invention. In practical applications, the specific number C of majority classes can be determined based on the class distribution of the actual speech dataset; this invention does not impose a limit on the specific number of classes. Furthermore, although this embodiment uses the ReLU activation function as an example in the classification mapper, this does not limit the scope of this invention. Those skilled in the art can replace ReLU with other activation functions with nonlinear mapping capabilities, such as GELU, Leaky ReLU, PReLU, or Swish, depending on the model's convergence or computational efficiency requirements.

[0032] In one embodiment of this application, the distillation feature characterizer preprocesses the original speech by including: To align the time dimension, zero-value padding is performed on the tail of the original speech frames with inconsistent lengths within the batch, and a Boolean mask is generated to mark the valid speech frames and the padding frames.

[0033] In one embodiment of this application, such as Figure 4 As shown, the raw audio is input through a mask and processed, then passes through a speech feature encoder. Here, we take the Wav2vec 2.0 model as an example. Each batch... Original audio clips with inconsistent lengths The speech feature vector for each sample is obtained through mask input preprocessing (determining the maximum temporal length of all samples in the current batch and padding samples with insufficient length with zeros to achieve uniform alignment of samples in the temporal dimension; at the same time, generating corresponding indicator masks, marking temporal positions containing valid information as 0 and zero-padding positions as 1) and then being processed by a speech feature encoder. At the same time, a Boolean mask corresponding to each sample is obtained. This is used to mark valid speech frames and padding frames in a sequence to eliminate interference from non-information segments. For the number of frames in the speech, is the hidden layer dimension of the feature vector.

[0034]

[0035] Then, a set of learnable vectors is introduced through the fixed-length speech representation generation module. ( (The number of learnable vectors) is used as the query vector. This batch Each sample speech feature matrix A three-dimensional tensor stacked along the batch dimension is processed by a learnable weight matrix. , , What is needed to acquire attention mechanism , , matrix:

[0036] To eliminate the interference of padding frames on feature aggregation, an additive mask aligned with the dimension of the attention score matrix is ​​constructed. Due to Boolean mask Time dimension only And attention score matrix The dimension is Therefore, it is necessary to Along the vector dimension Broadcasting is performed. The mapping strategy is as follows: [The remaining text appears to be incomplete and requires further context.] Positions marked as padding are mapped to negative infinity to force their probabilities to approach zero during Softmax normalization; positions marked as valid information are mapped to 0 to preserve the original semantic associations.

[0037] In the formula, The additive mask matrix represents the first... Line number The element values ​​of the column, where To query the vector index, The time frame index is then used. Subsequently, a multi-head cross-attention mechanism combined with additive masking is employed. By filtering out interference from padding frames when calculating weights, feature distillation is achieved.

[0038]

[0039]

[0040] In the formula, The dimension corresponding to each head in each multi-head attention. For the number of heads, This is the attention weight matrix. To output the projection matrix, This is a characteristic of distillation.

[0041] Distillation characteristics Compared with the original query Perform addition and layer normalization operation ( Obtain the Refined Feature :

[0042] Subsequently, a feedforward neural network containing two linear layers and the GELU activation function is used. Capture non-linear semantics to obtain the final features. :

[0043] Will Average pooling is then performed along the second dimension to generate a fixed-dimensional speech feature vector for final classification. :

[0044] It should be noted that, Figure 4 The feature encoder described uses the Wav2vec 2.0 model as an example to illustrate the process of speech feature extraction and representation. However, this is only an illustrative illustration and does not constitute a limitation on the scope of protection of this invention. In practical applications, the feature encoder is not limited to the Wav2vec 2.0 model. Those skilled in the art can choose Hubert, WavLM, or other pre-trained models or neural network architectures with corresponding feature extraction capabilities according to actual needs. This invention does not make any specific limitations in this regard.

[0045] In one embodiment of this application, the joint loss used in the majority category prototype benchmark construction model consists of classification loss, prototype cohesion loss and prototype separation regularization loss. The model parameters are jointly updated through backpropagation, so that similar features are aggregated and dissimilar prototypes are separated.

[0046] In one embodiment of this application, in order to establish a robust majority class prototype anchor in the feature space, the model is optimized using a joint loss function. This joint loss... Classification loss Prototype cohesion loss and the separation regularization loss between prototypes It consists of three parts:

[0047]

[0048]

[0049]

[0050] In the formula, It is a sample The true label, This is a regularization term introduced to increase the inter-class spacing. All the above loss functions are jointly updated through backpropagation, which brings similar features closer together while pushing away dissimilar prototypes, resulting in compact and well-separated clusters in the cosine space.

[0051] In one embodiment of this application, the minority class boundary reinforcement model includes a distillation feature characterizer and a minority class prototype anchor builder, wherein: S401: The distillation feature characterizer loads and freezes the encoder parameters of the majority class prototype benchmark model, and extracts minority class discriminative feature embeddings through a new set of learnable vectors.

[0052] S402: The minority class prototype anchor builder uses the exponential moving average algorithm to construct minority class prototype centers and combines the majority class prototype anchors to calculate the similarity between the sample and each class of prototypes.

[0053] In one embodiment of this application, such as Figure 5 As shown, the model consists of a distillation feature representation unit and a minority-class prototype anchor builder. The original audio is first input into the model's distillation feature representation unit. At this point, the speech feature encoder Wav2vec 2.0 model within the distillation feature representation unit uses the majority-class prototype benchmark to construct the optimal model parameters obtained after model training and freezes them. It is then trained with a new set of learnable vectors, outputting a fixed-dimensional speech feature vector. .

[0054] Construct a minority class prototype center and obtain minority class prototypes by weighting and fusing the feature mean of the current batch with historical states using an exponential moving average algorithm. Inherit from the first phase and lock the prototype matrix of the majority classes. As a reference anchor point. Through calculating feature embeddings. By using the scaling cosine similarity between the model centers (including the frozen majority class anchors and the updated minority class prototypes), a multi-level alignment relationship is established to achieve a preliminary definition of the distribution regions of different categories in a stable feature space.

[0055] In one embodiment of this application, the minority class prototype anchor builder extracts the maximum similarity between the test speech and all majority class prototype anchors using the maximum aggregation operator, which is used as the majority class aggregation score. At the same time, it calculates the similarity between the test speech and the minority class prototype center as the minority class prototype score, and completes the determination of minority class and non-minority class based on the two scores.

[0056] In one embodiment of this application, in order to accurately characterize the distinction boundary between the minority and majority classes, the maximum aggregation operator (max) is used to extract the similarity between the sample and all majority class anchor points, thereby obtaining the majority class aggregation score representing the global background reference. Simultaneously, the similarity between the sample and the dynamically updated minority class prototype is calculated to obtain the minority class prototype score. The calculation formula is:

[0057]

[0058] In the formula, Representing what was inherited from the first stage, representing the first Feature center vectors of most categories; The feature center vector represents the top minority class.

[0059] In one embodiment of this application, the joint loss used by the minority class boundary reinforcement model consists of binary cross-entropy loss and noise contrastive estimation loss. It updates the minority class-specific learnable vector and minority class prototype center parameters through backpropagation, thereby strengthening the discriminative boundary between the minority and majority classes.

[0060] In one embodiment of this application, to strengthen the discriminative boundary between the minority and majority classes while maintaining the stability of the majority class prototype topology, the system employs a hybrid objective function for optimization. This joint loss is composed of a binary cross-entropy loss. Loss estimation by comparison with noise It consists of two parts:

[0061]

[0062]

[0063] in, These labels are specifically designed for the second stage, with a value of 1 for the minority class and 0 for the majority class. All the loss functions mentioned above jointly update the learnable vectors specifically for the minority class through backpropagation, while simultaneously calibrating the prototype centers of the minority class through the EMA algorithm. Thus, without disrupting the existing clustering distribution of the majority class, the discriminative boundaries of the long-tail classes are accurately defined in the cosine space through asymmetric discriminative feature enhancement.

[0064] Figure 5The number of majority class background anchors shown is merely an example. Those skilled in the art should understand that the size of the majority class background anchor set corresponds to the total number of majority classes established in the first phase, and this number depends on the specific task settings and is not limited to the number shown in the figure.

[0065] S203: Perform joint training on the majority class prototype benchmark construction model and the minority class boundary reinforcement model, wherein the majority class prototype benchmark construction model is trained using only majority class speech samples, establishes a stable majority class prototype anchor point in the feature space and saves the optimal encoder parameters, and the minority class boundary reinforcement model is trained using the full amount of speech samples containing the minority class, and delineates the minority class decision boundary while maintaining the stability of the majority class distribution.

[0066] In this embodiment, the majority class prototype benchmark construction model is first trained to establish a stable head class prototype anchor point to construct a global benchmark. Then, the speech feature encoder and head class anchor point parameters in the distillation feature representation are locked, and the minority class prototype anchor point builder is activated only for minority class samples to achieve refined enhancement of the minority class decision boundary and global alignment of features of all classes.

[0067] In one embodiment of this application, the joint training of the majority class prototype benchmark construction model and the minority class boundary reinforcement model includes: S501: Select majority class speech samples to train the majority class prototype benchmark to build the model, and save the optimal model parameters and the convergent and stable majority class prototype anchor point.

[0068] S502: Initialize the minority class boundary reinforcement model, load and freeze the optimal model parameters and majority class prototype anchor points.

[0069] S503: Train a minority class boundary enhancement model using full speech samples containing minority classes, remap the original speech labels to binary labels for minority and non-minority classes, and update only the minority class-specific learnable vector and minority class prototype center parameters.

[0070] In one embodiment of this application, firstly, only speech samples belonging to the majority category in the dataset are selected as training inputs to build the majority category prototype benchmark model. The parameters of the distillation feature representation unit and the classification mapper are jointly updated using the backpropagation algorithm. Simultaneously, the prototype centers of each majority category are dynamically and iteratively updated in the feature space using the exponential moving average algorithm in the majority category prototype anchor builder. After training, the parameters of the majority category prototype benchmark model with the best classification performance, along with all convergent and stable majority category prototype centers, are selected and saved as global feature anchors for subsequent stages.

[0071] Next, the minority class boundary reinforcement model is initialized. The optimal speech feature encoder model parameters saved in the first stage are loaded to inherit the learned general speech feature representation capabilities. Simultaneously, all majority class prototype centers established in the first stage are loaded into the model as non-trainable "super prototype anchors." Parameter freezing is performed on the speech feature encoder and the aforementioned super prototype anchors to block their gradient backpropagation paths, ensuring that the distribution geometry of the majority classes in the feature space remains absolutely stable and does not drift during subsequent training.

[0072] Finally, a second stage of training is performed using the full dataset, including the minority class, and all original labels are remapped to binary opposition labels for "minority class" and "non-minority class". A new set of learnable vectors is initialized, and feature embeddings specifically for minority class discrimination are distilled from the frozen encoder output through a cross-attention mechanism. During training, the model only updates parameters for the newly introduced learnable vectors and minority class prototype centers. By minimizing the binary classification loss and the noise contrastive estimation loss, the model drives minority class features to aggregate towards their prototypes and maintains a maximum feature distance from the frozen majority class super prototype anchors. This allows for a refined reconstruction of the minority class decision boundary in a stable background space, preserving convergent and stable minority class prototype centers.

[0073] S204: Perform speech classification inference based on the trained model to obtain the classification result.

[0074] In one embodiment of this application, the step of performing speech classification inference based on the trained model to obtain the classification result includes: S601: Input the speech to be tested into the minority class boundary enhancement model and calculate the scaling cosine similarity between the speech features and the minority class prototype center and the majority class prototype anchor point.

[0075] S602: If the similarity of the minority class prototype is dominant, determine that the speech to be tested belongs to the minority class and output the result.

[0076] S603: Otherwise, input the speech to be tested into the majority class prototype benchmark to build a model, determine the specific category of the speech in the majority class and output the result.

[0077] In one embodiment of this application, hierarchical concatenation discrimination is performed to achieve the final multi-class recognition. The speech to be tested is input into the minority class boundary enhancement model constructed in the second stage. Discriminative features are extracted using the model's unique parameters, and the scaled cosine similarity between the features and the previously saved minority class prototype centers and majority class background anchor points is calculated. The similarity response values ​​of the sample with the minority class prototypes and the maximum similarity response value with all majority class background anchor points are compared. If the minority class prototype response is dominant, the sample is determined to fall within the minority class decision boundary, the corresponding minority class result is directly output, and the inference is terminated. If the initial judgment shows that the sample tends to be in the majority class background set, the majority class prototype benchmark construction model built in the first stage is activated. The same sample is then re-mapped and analyzed in multiple dimensions using this benchmark model. The prediction probability of the class mapper is combined with the matching degree of the benchmark prototype anchor point to make a joint decision. The final specific class classification is determined in the majority class subspace, thus completing the complete reasoning process from binary boundary filtering to multi-class fine recognition.

[0078] In the aforementioned unbalanced speech classification method based on prototype learning, original speech samples are obtained; a majority class prototype baseline construction model and a minority class boundary reinforcement model are constructed; joint training is performed on the majority class prototype baseline construction model and the minority class boundary reinforcement model. The majority class prototype baseline construction model is trained using only majority class speech samples, establishing stable majority class prototype anchors in the feature space and saving optimal encoder parameters. The minority class boundary reinforcement model is trained using all speech samples containing the minority class, defining the minority class decision boundary while maintaining the stability of the majority class distribution. Speech classification inference is performed based on the trained model to obtain the classification result. Compared with existing technologies, this invention achieves refined extraction and category recognition of speech features, effectively solving the problems of unstable classification results and performance degradation caused by extreme class imbalance and inconsistent speech duration, and improving the accuracy and robustness of speech classification in complex environments.

[0079] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0080] Based on the same inventive concept, this application also provides a prototype-based non-equilibrium speech classification device for implementing the prototype-based non-equilibrium speech classification method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations of one or more prototype-based non-equilibrium speech classification device embodiments provided below can be found in the limitations of the prototype-based non-equilibrium speech classification method described above, and will not be repeated here.

[0081] In one embodiment, such as Figure 6 As shown, a non-equilibrium distributed speech classification device 600 based on prototype learning is provided, including: a speech data acquisition module 601, a speech classification model construction module 602, a speech classification model training module 603, and a speech classification module 604, wherein: The voice data acquisition module 601 is used to acquire raw voice data and construct voice samples.

[0082] The speech classification model building module 602 is used to build a prototype baseline model for the majority class and a boundary reinforcement model for the minority class.

[0083] The speech classification model training module 603 is used to perform joint training on the majority class prototype baseline construction model and the minority class boundary reinforcement model. The majority class prototype baseline construction model is trained using only majority class speech samples to establish a stable majority class prototype anchor point in the feature space and save the optimal encoder parameters. The minority class boundary reinforcement model is trained using the full set of speech samples containing minority classes to define the minority class decision boundary while maintaining the stability of the majority class distribution.

[0084] The speech classification module 604 is used to perform speech classification inference based on the trained model to obtain the classification result.

[0085] In one embodiment of this application, the majority-class prototype benchmark construction model includes a distillation feature characterizer, a classification mapper, and a majority-class prototype anchor builder, wherein: The distillation feature representation extracts frame-level sequence representations after preprocessing the original speech, and generates fixed-dimensional speech feature embeddings through learnable vector groups and cross-attention mechanisms. The classification mapper performs non-linear remapping of the speech feature embeddings through multiple layers of linear blocks; The majority class prototype anchor builder uses an exponential moving average algorithm to dynamically update the central representation of the majority class, and combines scaled cosine similarity to calculate the loss to establish the majority class prototype anchor.

[0086] In one embodiment of this application, the distillation feature characterizer preprocesses the original speech by including: To align the time dimension, zero-value padding is performed on the tail of the original speech frames with inconsistent lengths within the batch, and a Boolean mask is generated to mark the valid speech frames and the padding frames.

[0087] In one embodiment of this application, the joint loss used in the majority category prototype benchmark construction model consists of classification loss, prototype cohesion loss and prototype separation regularization loss. The model parameters are jointly updated through backpropagation, so that similar features are aggregated and dissimilar prototypes are separated.

[0088] In one embodiment of this application, the minority class boundary reinforcement model includes a distillation feature characterizer and a minority class prototype anchor builder, wherein: The distillation feature characterizer loads and freezes the encoder parameters of the majority-class prototype benchmark model, and extracts minority-class discriminative feature embeddings through a new set of learnable vectors. The minority class prototype anchor builder uses an exponential moving average algorithm to construct minority class prototype centers and combines them with majority class prototype anchors to calculate the similarity between samples and prototypes of all classes.

[0089] In one embodiment of this application, the minority class prototype anchor builder extracts the maximum similarity between the test speech and all majority class prototype anchors using the maximum aggregation operator, which is used as the majority class aggregation score. At the same time, it calculates the similarity between the test speech and the minority class prototype center as the minority class prototype score, and completes the determination of minority class and non-minority class based on the two scores.

[0090] In one embodiment of this application, the joint loss used by the minority class boundary reinforcement model consists of binary cross-entropy loss and noise contrastive estimation loss. It updates the minority class-specific learnable vector and minority class prototype center parameters through backpropagation, thereby strengthening the discriminative boundary between the minority and majority classes.

[0091] In one embodiment of this application, the joint training of the majority class prototype benchmark construction model and the minority class boundary reinforcement model includes: Select majority class speech samples to train the majority class prototype benchmark to build the model, and save the optimal model parameters and the convergent and stable majority class prototype anchor point; Initialize the minority class boundary reinforcement model, load and freeze the optimal model parameters and the majority class prototype anchor point; A minority class boundary enhancement model is trained using a full set of speech samples containing minority class data. The original speech labels are remapped to binary labels for both minority and non-minority classes, and only the minority class-specific learnable vector and minority class prototype center parameters are updated.

[0092] In one embodiment of this application, the step of performing speech classification inference based on the trained model to obtain the classification result includes: Input the speech to be tested into the minority class boundary enhancement model and calculate the scaling cosine similarity between the speech features and the minority class prototype center and the majority class prototype anchor point. If the similarity to the minority class prototype is dominant, the speech to be tested is determined to be a minority class and the result is output. Otherwise, the test speech is input into the majority class prototype baseline to build a model, determine the specific category of the test speech in the majority class, and output the result.

[0093] Each module in the aforementioned unbalanced speech classification device based on prototype learning can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0094] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a non-equilibrium speech classification method based on prototype learning. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0095] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0096] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0097] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0098] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0099] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0100] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0101] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A non-equilibrium speech classification method based on prototype learning, characterized in that, The method includes: Obtain the original speech and construct speech samples; Construct a prototype benchmark model for the majority of categories and a boundary reinforcement model for the minority of categories; The majority class prototype baseline construction model and the minority class boundary reinforcement model are jointly trained. The majority class prototype baseline construction model is trained using only majority class speech samples to establish a stable majority class prototype anchor point in the feature space and save the optimal encoder parameters. The minority class boundary reinforcement model is trained using the full set of speech samples containing minority classes to define the minority class decision boundary while maintaining the stability of the majority class distribution. Based on the trained model, speech classification inference is performed to obtain the classification result.

2. The non-equilibrium speech classification method based on prototype learning according to claim 1, characterized in that, The majority-class prototype benchmark construction model includes a distillation feature characterizer, a classification mapper, and a majority-class prototype anchor builder, wherein: The distillation feature representation extracts frame-level sequence representations after preprocessing the original speech, and generates fixed-dimensional speech feature embeddings through learnable vector groups and cross-attention mechanisms. The classification mapper performs non-linear remapping of the speech feature embeddings through multiple layers of linear blocks; The majority class prototype anchor builder uses an exponential moving average algorithm to dynamically update the central representation of the majority class, and combines scaled cosine similarity to calculate the loss to establish the majority class prototype anchor.

3. The non-equilibrium speech classification method based on prototype learning according to claim 2, characterized in that, The distillation feature characterizer preprocesses the original speech, including: To align the time dimension, zero-value padding is performed on the tail of the original speech frames with inconsistent lengths within the batch, and a Boolean mask is generated to mark the valid speech frames and the padding frames.

4. The non-equilibrium speech classification method based on prototype learning according to claim 2, characterized in that, The joint loss used in the multi-category prototype benchmark construction model consists of classification loss, prototype cohesion loss, and prototype separation regularization loss. The model parameters are jointly updated through backpropagation, so that similar features are aggregated and dissimilar prototypes are separated.

5. The non-equilibrium speech classification method based on prototype learning according to claim 1, characterized in that, The minority class boundary reinforcement model includes a distillation feature characterizer and a minority class prototype anchor builder, wherein: The distillation feature characterizer loads and freezes the encoder parameters of the majority-class prototype benchmark model, and extracts minority-class discriminative feature embeddings through a new set of learnable vectors. The minority class prototype anchor builder uses an exponential moving average algorithm to construct minority class prototype centers and combines them with majority class prototype anchors to calculate the similarity between samples and prototypes of all classes.

6. The non-equilibrium speech classification method based on prototype learning according to claim 5, characterized in that, The minority class prototype anchor builder extracts the maximum similarity between the test speech and all majority class prototype anchors using the maximum aggregation operator, which is used as the majority class aggregation score. At the same time, it calculates the similarity between the test speech and the minority class prototype center as the minority class prototype score. Based on the two scores, the determination of minority class and non-minority class is completed.

7. The non-equilibrium speech classification method based on prototype learning according to claim 5, characterized in that, The joint loss used in the minority class boundary strengthening model consists of binary cross-entropy loss and noise contrastive estimation loss. It strengthens the discriminative boundary between the minority class and the majority class by updating the minority class-specific learnable vector and the minority class prototype center parameter through backpropagation.

8. The non-equilibrium speech classification method based on prototype learning according to claim 1, characterized in that, The joint training of the majority-class prototype benchmark model and the minority-class boundary reinforcement model includes: Select majority class speech samples to train the majority class prototype benchmark to build the model, and save the optimal model parameters and the convergent and stable majority class prototype anchor point; Initialize the minority class boundary reinforcement model, load and freeze the optimal model parameters and the majority class prototype anchor point; A minority class boundary enhancement model is trained using a full set of speech samples containing minority class data. The original speech labels are remapped to binary labels for both minority and non-minority classes, and only the minority class-specific learnable vector and minority class prototype center parameters are updated.

9. The non-equilibrium speech classification method based on prototype learning according to claim 1, characterized in that, The classification results obtained by performing speech classification inference based on the trained model include: Input the speech to be tested into the minority class boundary enhancement model and calculate the scaling cosine similarity between the speech features and the minority class prototype center and the majority class prototype anchor point. If the similarity to the minority class prototype is dominant, the speech to be tested is determined to be a minority class and the result is output. Otherwise, the test speech is input into the majority class prototype baseline to build a model, determine the specific category of the test speech in the majority class, and output the result.

10. A non-equilibrium speech classification device based on prototype learning, characterized in that, The device includes: The voice data acquisition module is used to acquire raw voice data and construct voice samples. The speech classification model building module is used to build a prototype baseline model for the majority class and a boundary reinforcement model for the minority class. The speech classification model training module is used to perform joint training on the majority class prototype baseline construction model and the minority class boundary reinforcement model. The majority class prototype baseline construction model is trained using only majority class speech samples, establishes a stable majority class prototype anchor point in the feature space and saves the optimal encoder parameters, and the minority class boundary reinforcement model is trained using the full set of speech samples containing minority classes, and delineates the minority class decision boundary while maintaining the stability of the majority class distribution. The speech classification module is used to perform speech classification inference based on the trained model to obtain the classification result.

Images