Sensitive speech recognition model construction method, device and medium

Abstract

The application provides a sensitive speech recognition model construction method, device and system, wherein the method comprises the following steps: collecting untagged data and data with sensitive labels; using an unsupervised learning strategy, setting an enhanced semantic and context module and a double balance loss function module to construct a sensitive speech recognition model; fine-tuning the sensitive speech recognition model to obtain a trained sensitive speech recognition model; and verifying the effectiveness of the trained sensitive speech recognition model by using the data with sensitive labels. The double balance loss function fully considers the problems of sample difficulty imbalance and category imbalance, so as to improve the recognition performance of the sensitive speech recognition model.

Description

Technical Field

[0001] This document relates to the field of sensitive speech recognition technology, and in particular to a method, device and system for constructing a sensitive speech recognition model. Background Technology

[0002] Significant progress has been made in the automatic identification of sexist language. Most excellent studies have employed advanced deep learning models based on Transformers, resulting in substantial improvements in recognition performance. However, only a few studies have focused on the theoretical, domain-specific, categorical, labeled, and distributed aspects of sexist texts. These cross-disciplinary studies across deep learning have been severely neglected, yet they are crucial for improving the generalization performance of models. Current problems mainly focus on the following aspects:

[0003] (1) The importance of domain knowledge in social media has been overlooked. Research shows that as the influence of social media expands, most data containing sexist text originates from social media, and its impact in this domain is becoming increasingly severe. However, existing deep learning methods often use pre-trained models with a relatively sparse proportion of domain knowledge when dealing with natural language processing tasks. This results in pre-trained models lacking strong semantic and encoding capabilities to accurately process sexist samples, thus failing to extract features from sexist text and limiting the performance of downstream classifiers.

[0004] (2) The importance of relevant psychological theories has been overlooked. The "anthropomorphism theory" in social prejudice studies posits that when people hold negative stereotypes about a group, they are more likely to attribute the group's behavior or characteristics to internal factors. In this theory, "discrimination anchors" refer to specific characteristics or behaviors that indicate prejudice or negative stereotypes when dealing with a particular group. Cryan et al. enhanced linguistic features by artificially constructing prejudice or emoji word lists, thus linking deep learning with prejudice studies theory, and their results showed a significant improvement in model quality. However, similar work is scarce, and Cryan et al.'s method has limitations, failing to automate the process and neglecting important features such as semantics and context.

[0005] (3) Lack of research on the uneven distribution and difficulty of gender discrimination sample categories. Recent work shows that gender bias samples are unevenly distributed across categories in the real world, and this problem exists in most datasets. Furthermore, with the continuous development of social media networks, there is a need to refine the granularity of gender discrimination categories to improve the interpretability, trust, and understanding of decisions made using automated tools, which adds considerable difficulty to classification models. Many methods rely solely on language models with more parameters to address this issue, but the performance improvement is often very weak, and they also suffer from drawbacks such as high memory consumption and slow inference speed. Summary of the Invention

[0006] This invention provides a method, device, and system for constructing a sensitive speech recognition model, aiming to solve the above-mentioned problems.

[0007] This invention provides a method for constructing a sensitive speech recognition model, characterized by comprising:

[0008] S1. Collect unlabeled data and data with sensitive labels;

[0009] S2. An unsupervised learning strategy is adopted, and modules for enhancing semantics and context and a dual-balance loss function module are set up to construct a sensitive speech recognition model.

[0010] S3. Fine-tune the sensitive speech recognition model to obtain a trained sensitive speech recognition model;

[0011] S4. Verify the effectiveness of the trained sensitive speech recognition model using data with sensitive labels.

[0012] This invention also provides an electronic device, comprising:

[0013] Processor; and,

[0014] A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the sensitive speech recognition model construction method described above.

[0015] This invention also provides a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the sensitive speech recognition model construction method described above.

[0016] The present invention has the following beneficial effects:

[0017] The sensitive speech recognition model construction method disclosed in this invention first employs an unsupervised learning strategy to enhance the depth of the pre-trained language model in understanding biased samples, thereby enhancing knowledge of the social media domain. Then, by enhancing semantics and context, the model's global attention to discriminatory anchors in gender-biased text is improved. Finally, the sensitive speech recognition model construction method alleviates the problems of class imbalance and difficulty imbalance in the large number of gender-biased samples on current social media platforms.

[0018] In the comparative experiments during the validation phase, the proposed method significantly outperformed several powerful baseline models, such as DeBERTa-v3-base, RoBERTa-base, and RoBERTa-large, in all three sub-tasks. This result not only fully validates the effectiveness of the sensitive speech recognition model construction method provided in this embodiment of the invention, but also demonstrates that starting from the perspective of gender discrimination theory and data distribution can stimulate the model's ability to perform the challenging task of gender discrimination recognition, while improving the interpretability of automated model decisions.

[0019] In the official evaluation phase experiments, the statistical indicators demonstrated, such as the first quartile, mean, median, and third quartile, also showed significant technical advantages in terms of statistical ranking. These indicators collectively demonstrate the multifaceted advantages of the sensitive speech identification model construction method disclosed in this application, providing a strong reference for subsequent research. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a sensitive speech recognition model construction method according to an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of a sensitive speech recognition model according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0024] Method Implementation Examples

[0025] This invention provides a method for constructing a sensitive speech recognition model. Figure 1 This is a flowchart of a sensitive speech recognition model construction method according to an embodiment of the present invention. Figure 1As shown, the sensitive speech recognition model construction method of this invention includes:

[0026] S1. Collect unlabeled data and data with sensitive labels. After collecting the data, clean the unlabeled data and divide it into training set and validation set. Divide the data with sensitive labels into three sub-task datasets with increasing task size. Divide each sub-task dataset into training set, validation set and test set according to a predetermined ratio.

[0027] The unlabeled data collected in this embodiment of the invention comes from two publicly available English social media datasets, Gab and Reddit, each containing 1 million samples: gab_1M_unlabeled and reddit_1M_unlabeled. After data collection, the unlabeled data is cleaned, including removing emojis, URLs, and HTML, and converting abbreviations. The unlabeled dataset is then divided into training and validation sets to facilitate subsequent unsupervised training.

[0028] This invention collects labeled discrimination identification datasets characterized by multi-classification, multi-granularity, strong representativeness, and high effectiveness. A labeled sexism dataset, EDOS, with 20,000 entries is collected, including 10,000 from Reddit and 10,000 from Gab. This dataset labels what constitutes sexist content and explains why it is sexist, which improves the interpretability, trust, and understanding of automated model decisions.

[0029] The EDOS dataset was then classified in detail into three categories: binary task A (predicting whether a post contains gender discrimination), four-category task B (predicting one of four categories: threat, denigration, hostility, and prejudice), and eleven-category task C. The ratio of training set, validation set, and test set for the three tasks was 70%, 10%, and 20%, respectively. Label distribution analysis was then performed on the three sub-task datasets.

[0030] S2. An unsupervised learning strategy is adopted, and modules for enhancing semantics and context and a dual-balance loss function module are set up to construct a sensitive speech recognition model. Figure 2 This is a schematic diagram of a sensitive speech recognition model according to an embodiment of the present invention. Figure 2 As shown, S2 specifically includes:

[0031] S21. Use unlabeled data as unsupervised training corpus for the pre-trained model, and perform unsupervised training using a masked language model to obtain vector sequence B;

[0032] In this embodiment of the invention, a pre-trained model BERTweet-large containing 850 million Tweet social media samples in the pre-training stage is used. The gab_1M_unlabelled and reddit_1M_unlabelled processed in S1 are used as unsupervised training corpora for the BERTweet-large model. This approach can enhance the model's sensitivity to multi-source social media domain knowledge and accelerate the transfer of unsupervised domain knowledge.

[0033] Unsupervised training using only the masked language model MLM is performed as follows:

[0034] A portion of the words in the input sentence, for example, 15% of the words, will be randomly replaced with a special masking symbol, such as "[MASK]";

[0035] The objective function of the masked language model MLM can be expressed as maximizing the log-likelihood probability of the masked word, such as L(θ; X, M) = ∑ i∈M logP(w i |w ～i As shown in the figure; where the sentence X=(w1,w2,...,w) N A subset of words is masked, forming a masking set M; θ represents the model parameters; P(w i |w ～i ,θ) is given model parameters and divided by w i Given all other words, predict the correct masked word w i The probability of w ～i This represents the set of all words except the i-th word;

[0036] We used the transformers library provided by HuggingFace and imported the TrainingArguments package. The training parameters were configured as follows: batch_size was set to 16 or 32; the initial learning rate was set to 2e-5, and a warmup strategy was used to gradually increase the learning rate before decaying it; the training epoch was set to 10, and training was performed on a Tesla P100.

[0037] Import the Trainer package, integrate the BERTweet-large model parameters, Tokenizer, training parameters, dataset, etc. into the Trainer object, and call the train() method to officially start unsupervised training.

[0038] The complete unsupervised training process is as follows: A sentence X = (w1, w2, ..., w NThe input is fed into a pre-trained model adapted to the social media domain, abbreviated as LLM_DA. The model generates a vector representation for each word, resulting in a vector sequence B = (b1, b2, ..., b...). n ), that is, B = LLM_DA(X), where w i These are the words in the sentence, and N is the number of words.

[0039] S22. Process the vector sequence B using LSTM and Attention mechanisms to enhance the attention to discriminatory anchors in sensitive statements;

[0040] This invention enhances the model's attention to "discriminatory anchors" in biased texts, such as gender, age, religion, and race, by designing a module based on enhanced semantics and context, as follows:

[0041] First, to better understand sentence structure and word order information of "discrimination anchors," a recurrent neural network (LSTM) is introduced, which is effective in processing sequential data and has advantages in handling long-distance dependencies. Specifically, the vector sequence B output by LLM_DA is input into an LSTM network. The LSTM network processes the entire sequence and outputs the final hidden state H, which represents the entire sentence H = LSTM(B).

[0042] Secondly, the introduced Attention mechanism allows for the assignment of different weights to each word in the output. This helps the model focus on the words that contribute most to the meaning of the sentence, while also better capturing the importance of "discrimination anchors." Specifically, for each vector b output by LLM_DA... i The Attention mechanism is applied, which typically involves weight calculation, using a Softmax function to compute the weight 'a' for each word. i =Softmax(W a b i +b a ), where W a and b a These are the parameters of the Attention layer. Ultimately, the weights are multiplied by the word vector, and then summed to obtain the sentence vector.

[0043] Finally, by using Concat, the outputs of Attention and LSTM pooling are fused together to obtain V = Concat(A,H). This helps to capture the "discrimination anchor" representations that the pre-trained model cannot notice. Such diversity can improve the model's adaptability to different types of text.

[0044] S23. Use the improved cross-entropy loss function and combine it with the dual Dropout algorithm for data augmentation.

[0045] This invention employs an improved cross-entropy loss function, which balances both the problem of uneven distribution of sample categories and the problem of imbalance between easy and difficult samples. Furthermore, it combines double Dropout for data augmentation to improve the model's generalization ability. Specifically, this includes:

[0046] The sentence vector V obtained from S22, which incorporates multi-granularity features, is used as the input to a single-layer feedforward neural network (FNN). Then, the output of the FNN is processed through the Dropout algorithm and a Softmax layer to obtain the normalized probability distribution P(y|x).

[0047] In this embodiment of the invention, to improve the generalization performance of the model, make it better adaptable to different data distributions and input variations, and enhance robustness, the above steps can be repeated twice to obtain two different probability distributions p1(y). i |x i ) and p2(y i |x i Because the Dropout algorithm is random, the two probability distributions are not exactly the same. This process is similar to training the model multiple times, and can be seen as model ensemble or data augmentation, i.e., p1(y i |x i ),p2(y i |x i =Softmax(Dropout) random (FNN(V)));

[0048] The core objectives of the balanced loss function are: 1) to make the different outputs after the Dropout algorithm as consistent as possible, thus achieving data expansion; 2) to mitigate the impact of sample imbalance on training; and 3) to reduce the problems caused by sample class imbalance. Therefore, the balanced loss function consists of three parts.

[0049] The first part uses KL divergence to measure the difference between two probability distributions. Following the approach of SimCSE, data augmentation is implemented to ensure that the outputs of different Dropout models are as consistent as possible, with a loss L... KL (p1, p2) is defined as follows:

[0050]

[0051] The second part, drawing on the idea of ​​FocalLoss, aims to improve the performance of gender discrimination detection tasks by reducing the focus on easily classified samples and allowing the model to concentrate more on difficult-to-classify samples. For easily classified samples, a (1-p) t ) γ The terms, where p tThe predicted probability γ of the model for a sample is an adjustment factor that controls the weight of easily classified samples. When γ > 0, the weight of easily classified samples is reduced, and the learning of difficult-to-classify samples is strengthened. When γ is 0, it degenerates into the cross-entropy loss function, and this part of the loss function L... F (p t The definition is as follows:

[0052] L F (p t )=-(1-p t ) γ *log(p t );

[0053] The third part involves statistically analyzing the number of samples (n) for each class in the dataset before training. This aims to encourage the model to focus more on the class with fewer samples, and a balanced weighting is designed. Wherein, β is a hyperparameter, controlled in the range of 0.9 to 0.99. When n increases, this weight tends to increase rapidly.

[0054] Finally, the complete dual-balance loss function is defined as follows, where α is the weight hyperparameter of the first part of the KL divergence loss;

[0055]

[0056] S3. Fine-tune the sensitive speech recognition model to obtain a trained sensitive speech recognition model; S3 specifically includes:

[0057] This invention uses the Transformers library of huggingface and PyTorch to build the network architecture and training code for a sensitive speech recognition model;

[0058] Using the AdamW optimizer, the weight decay is set to 0.01, eps is 1e-8, the learning rate is 2e-5, and the batch size is 16.

[0059] A linear learning rate scheduler is used for warm-up, with a warm-up ratio of 0.2.

[0060] Training was performed on a Tesla P100 GPU;

[0061] Stratified 5-fold cross-validation was used for each experiment to evaluate model performance and hyperparameter selection. The experimental results shown are the average of the results from 5 randomized seeds.

[0062] S4. Validate the effectiveness of the trained sensitive speech recognition model using data with sensitive labels; S4 specifically includes:

[0063] Load the optimal model parameter information saved during the training phase and initialize the sensitive speech recognition model.

[0064] Load the three sub-datasets with increasing task size respectively;

[0065] Start making predictions and calculate the Macro-F1 metric for the model;

[0066] Evaluate the performance differences compared to other baseline models;

[0067] Submit the integrated model to the official testing website for authoritative testing and statistical ranking of model performance.

[0068] Akshita Jha discusses the use of Twitter data for discriminatory analysis and classification, particularly regarding two forms of discrimination: hostility and benevolence. They used an approach based on ambivalent sexism theory to label discriminatory speech in Twitter data; they also employed machine learning algorithms and natural language processing techniques to analyze and classify these speech. Specifically, they used techniques such as the bag-of-words model and support vector machines to train classifiers and evaluated their performance on a test set. Anzovino et al. conducted the first investigation into automatic misogyny detection in social media; Peiling Yi et al. addressed the limitation of Transformer models, which only accept inputs of finite length, by proposing a framework called LS-CB to detect cyberbullying from lengthy social media conversations. Nozza et al. attempted to measure and mitigate unintended biases in machine learning models used for misogyny detection, conducting extensive misogyny detection in multilingual and cross-domain scenarios. A team from Oxford University contributed to the development of an automatic sexism detection system by publishing a dataset and analysis of interpretable online sexism detection, along with an accompanying SemEval task submission, which is more accurate and interpretable through fine-grained labeling. These contributions mark a step towards a safer internet for women.

[0069] In recent years, there have been many innovative research results in this field in China. Aiqi Jiang et al. made the first attempt to create a Chinese sexism dataset and provide a benchmark, including implementations of various machine learning and deep learning methods. Yimeng Gu proposed an effective multimodal multitask variational autoencoder (MMVAE) to detect malicious memes containing information unfriendly to women and identify their fine-grained categories. This method can solve the challenge of common representation of multimodal features and achieves high F1 scores on SemEval-2022 MAMI subtasks A and B. Due to the lack of reliable datasets in Chinese scenarios, Jiawen Deng proposed a benchmark for Chinese offensive language analysis—COLD, which includes a Chinese offensive language dataset—COLDATASET and a baseline detector—COLDETECTOR trained on this dataset. Xiaojun Rao et al. designed the Transformer-based Chinese pre-trained model RoBERTa-CHHSD to extract deeper hate speech features from text and classify Chinese hate speech.

[0070] Based on the analysis of the current state of research, significant progress has been made in the field of automatic sexist speech recognition. Most excellent studies have employed advanced deep learning models based on Transformers, resulting in substantial improvements in recognition performance. However, only a few studies have focused on related issues such as the theory, domain, category, labeling, and distribution of sexist texts. These cross-disciplinary studies across deep learning have been severely neglected, yet they are crucial for improving the generalization performance of models.

[0071] Therefore, this invention analyzes the theoretical frameworks of social role theory and gender difference theory, and proposes a discriminatory speech detection method based on unsupervised domain enhancement and dual-balance loss. This method can compensate for the sparse domain knowledge of pre-trained models, strengthen the bridge between psychology and deep learning, and alleviate the problems of class imbalance and difficulty imbalance in a large number of gender discrimination samples, thereby significantly improving the model's performance on difficult datasets and enhancing recognition performance.

[0072] By employing the embodiments of the present invention, the following beneficial effects are achieved:

[0073] This invention discloses a discriminatory speech identification method based on unsupervised domain enhancement and a dual-balance loss function. First, an unsupervised learning strategy is employed to enhance the depth of the pre-trained language model in understanding biased samples, thereby enhancing its knowledge of the social media domain. Then, by enhancing semantics and context, the model's global attention to "discriminatory anchors" in gender-biased text is increased. Finally, to alleviate the problems of class imbalance and difficulty imbalance in the large number of gender-biased samples on social media, a dual-balance loss function is proposed.

[0074] This invention presents detailed analyses and experiments on three publicly available EDOS datasets with varying classification difficulties. In comparative experiments during the validation phase, the method provided by this invention significantly outperformed several powerful baseline models, such as DeBERTa-v3-base, RoBERTa-base, and RoBERTa-large, in all three sub-tasks. This result not only fully validates the effectiveness of the proposed method but also demonstrates that addressing gender discrimination theory and data distribution can enhance the model's ability to perform the challenging task of gender discrimination identification, while simultaneously improving the interpretability of automated model decisions.

[0075] The sensitive speech recognition model proposed in this invention demonstrated significant technical advantages in statistical indicators, such as the first quartile, mean, median, and third quartile, during the official evaluation phase experiments. These indicators collectively prove the advantages of this invention in multiple aspects and provide a strong reference for subsequent research.

[0076] Device Example 1

[0077] An electronic device is provided according to an embodiment of the present invention, comprising:

[0078] Processor; and,

[0079] A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the sensitive speech recognition model construction method as described in the above method embodiments.

[0080] Device Example 2

[0081] This invention also provides a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the sensitive speech recognition model construction method as described in the above method embodiments.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a sensitive speech recognition model, characterized in that, include: S1. Collect unlabeled data and data with sensitive labels; S2. An unsupervised learning strategy is adopted, and modules for enhancing semantics and context and a dual-balance loss function module are set up to construct a sensitive speech recognition model. S3. Fine-tune the sensitive speech recognition model to obtain a trained sensitive speech recognition model; S4. Verify the effectiveness of the trained sensitive speech recognition model using the data with sensitive labels; S2 specifically includes: S21. Use the unlabeled data as the unsupervised training corpus for the pre-trained model, and perform unsupervised training using a masked language model to obtain vector sequence B; S22. Process the vector sequence B using LSTM and Attention mechanisms to enhance the attention to discriminatory anchors in sensitive statements; S23. Data augmentation is performed by using an improved cross-entropy loss function combined with a dual Dropout algorithm. In step S21, an unsupervised training using a masked language model is employed to obtain the vector sequence B, which specifically includes: A specific proportion of words in the input sentence are replaced with a masking symbol using a random function; The objective function of the masked language model is expressed as: Official 1; in, This represents the input sentence. Let X represent the N words, and M represent the occlusion set. Indicates model parameters, Given model parameters and excluding Given all other words, predict the correct masked word. The probability of; Indicates except the first The set of all words except for the first word; Set the model training parameters to generate a vector representation for each word in the input sentence, resulting in a vector sequence B; S22 specifically includes: The vector sequence B is input into the LSTM network. The LSTM network processes the entire sequence and outputs the final hidden state. ; For each vector in vector sequence B application Attention The mechanism calculates the weight of each word. , thus obtaining sentence vectors ; Where n is the number of words in vector sequence B; The Attention and LSTM pooling outputs are combined using Concat to obtain a vector. .

2. The method according to claim 1, characterized in that, S23 specifically includes: vector By repeatedly inputting the data into a single-layer feedforward neural network, two different probability distributions are obtained through the Dropout algorithm and a Softmax layer. and ; A dual-balance loss function is constructed by considering the imbalance between sample difficulty and class imbalance.

3. The method according to claim 2, characterized in that, The construction of a dual-balanced loss function by considering both sample imbalance and class imbalance specifically includes: KL divergence is used to measure the difference between two probability distributions. and Measurement of the difference between them, loss The definition is as follows: Official 2; in D KL This represents the KL divergence between two probability distributions; Adopting the concept of FocalLoss, a moderating factor is introduced. loss function The definition is as follows: Official 3; Design balancing weights ,in For hyperparameters, nc The number of each category in the sample; Through loss Loss function And constructing a dual-balance loss function using balanced weights : Official 4; in, For loss The weight hyperparameter.

4. The method according to claim 1, characterized in that, S1 further includes: cleaning the unlabeled data and dividing it into a training set and a validation set; dividing the data with sensitive labels into three sub-task datasets with increasing task size; and dividing each sub-task dataset into a training set, a validation set, and a test set according to a predetermined ratio.

5. The method according to claim 4, characterized in that, S4 specifically includes: S41. Load the trained sensitive speech recognition model and perform initialization operations; S42. Load the three subtask datasets of the increasing task order respectively; S43. Use the trained sensitive speech recognition model to make predictions and record the model metrics; S44. Compare the model indicators with the comparison model and analyze the comparison results.

6. An electronic device, comprising: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the sensitive speech recognition model construction method as described in any one of claims 1-5.

7. A storage medium for storing computer-executable instructions, which, when executed, implement the steps of the sensitive speech recognition model construction method as described in any one of claims 1-5.

Images