Harmful speech detection method and equipment based on robust knowledge distillation and medium

Abstract

The invention relates to a harmful speech detection method and device based on robust knowledge distillation, and a medium, and a training method of a used student model comprises the steps: carrying out the recognition and elimination of original speech data, and obtaining denoised speech data; randomly performing character-level adversarial attack on part of samples in the de-noised speech data to generate disturbance speech data so as to obtain an enhanced speech data set; word segmentation is carried out on the speech in the enhanced speech data set; randomly replacing marks in the teacher word segmentation sequence and the student word segmentation sequence through multiple Bernoulli tests to generate a disturbance word segmentation sequence; calculating a first loss value between the model prediction and the hard tag in the teacher model, and optimizing teacher model parameters; and calculating a second loss value in the student model based on the prediction probability of the student model, the hard tag and the soft tag output by the teacher model, and optimizing the student model parameters. Compared with the prior art, the method has the advantages of high accuracy, strong robustness, low time delay and the like.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and cybersecurity technology, and in particular to a method, device and medium for detecting harmful speech based on robust knowledge distillation. Background Technology

[0002] With the rapid development of social media, people have gained convenience in information exchange and expression of opinions, but at the same time, they also face the serious challenge of the widespread dissemination of harmful speech. Harmful speech usually manifests as insults, threats, discrimination, violence, and other forms of content, which not only damage individual mental health but also negatively impact the public environment. Therefore, establishing an efficient and accurate mechanism for detecting harmful speech has become an important and urgent task in cyberspace governance.

[0003] With the continuous advancement of natural language processing technology, AI-based content detection methods have gradually become the mainstream approach to combating harmful speech. Early methods primarily relied on rule matching or traditional machine learning models, but these methods have significant limitations in deep semantic understanding. In recent years, detection methods based on fine-tuning pre-trained language models have shown stronger semantic representation capabilities, improving detection performance. However, these models typically have a large parameter scale, resulting in slow inference speeds and making it difficult to meet the stringent response efficiency requirements of real-time detection scenarios.

[0004] To improve inference efficiency while maintaining detection performance, knowledge distillation, as an effective model compression technique, has been introduced into the harmful speech detection task. This method transfers knowledge from a large teacher model to a lightweight student model to achieve rapid inference in resource-constrained environments. However, existing knowledge distillation-based detection methods still have significant shortcomings: social media texts often contain a large amount of non-textual noise, such as garbled characters, usernames, URLs, and special symbols. After word segmentation, this noise introduces irrelevant features, interfering with the model's understanding of the core semantics, thus affecting the model's generalization ability and robustness. Furthermore, harmful speech data generally suffers from class imbalance, leading to insufficient learning of minority class samples by the model, further reducing the accuracy and stability of detection. Therefore, how to effectively overcome the challenges posed by noise interference and uneven data distribution while ensuring detection efficiency, and improve the robustness and adaptability of harmful speech detection models, is a technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a method, device and medium for detecting harmful speech based on robust knowledge distillation. Based on the linguistic features of the original speech, targeted denoising is performed, and a robust knowledge distillation process is constructed by combining character-level and tag-level dual-round data augmentation strategies. This effectively overcomes the problems of noise interference and data imbalance while compressing the model to improve inference speed, and ultimately enables the lightweight student model to have both high accuracy and strong generalization ability in the detection of harmful speech.

[0006] The objective of this invention can be achieved through the following technical solutions: According to one aspect of the present invention, a method for detecting harmful speech based on robust knowledge distillation is provided. The method is based on a student model obtained through robust knowledge distillation, and the specific steps of training the student model include: S1. Collect raw speech data, and through text parsing operations based on regular expressions, identify and remove unnecessary or redundant noise elements in the raw speech data to obtain denoised speech data. S2. Randomly select a portion of the denoised speech data to perform character-level adversarial attacks, generate perturbation speech data, and merge the perturbation speech data with the denoised speech data to form an enhanced speech dataset; S3. Use the word segmenters corresponding to the teacher model and the student model to segment the speech in the enhanced speech dataset to obtain the teacher segmentation sequence and the student segmentation sequence respectively; based on the preset perturbation probability, randomly replace the markers in the teacher segmentation sequence and the student segmentation sequence through multiple Bernoulli trials to generate a perturbation segmentation sequence; S4. Input the teacher perturbation sequence in the perturbation segmentation sequence into the teacher model, calculate the first loss value between the model prediction and the hard label based on the focus loss function, and optimize the teacher model parameters. S5. Input the student perturbation sequence in the perturbation segmentation sequence into the student model. Calculate the second loss value based on the predicted probability of the student model, the hard label, and the soft label output by the teacher model using a combined loss function. Optimize the student model parameters to obtain a trained student model. The combined loss function includes a student loss term based on the focus loss function and a distillation loss term based on the soft label of the teacher model.

[0007] Furthermore, the text parsing operations used for noise reduction in S1 include garbled character repair, removal of special symbols, URL removal, duplicate character elimination, username removal, and topic tag removal.

[0008] Furthermore, the specific steps of the character-level adversarial attack in S2 include: Based on the preset perturbation ratio and the amount of data in the denoised speech data, the number of adversarial attack rounds is determined, i.e., the number of perturbation samples to be generated. From the denoised speech data, each round of attack randomly selects a speech sample using a uniform distribution; for each selected speech sample, an adversarial attacker performs character-level adversarial attack operations with a preset perturbation probability, including random character replacement, deletion, insertion, and swapping of adjacent characters, to generate a perturbed speech sample; After completing all rounds of adversarial attacks, the perturbed speech samples generated in each round are combined to obtain the perturbed speech data.

[0009] Furthermore, in S3, the specific steps of randomly replacing the teacher's word segmentation sequence and the student's word segmentation sequence through multiple Bernoulli trials include: For each token in each teacher's word segmentation sequence and student's word segmentation sequence, perform a Bernoulli trial; if the Bernoulli trial fails, replace the current token with a predefined unknown token; if the Bernoulli trial succeeds, keep the current token unchanged. After traversing the entire teacher segmentation sequence and student segmentation sequence, the corresponding perturbed segmentation sequence is obtained.

[0010] Furthermore, in step S4, the specific steps for optimizing the teacher model parameters include: The teacher perturbation sequence in the perturbation segmentation sequence is input into the teacher model, and forward calculation is performed through the embedding layer and multiple Transformer layers to obtain the corresponding log odds; The logarithmic probability is normalized using a sigmoid function to obtain the teacher model's predicted probability for the current batch of samples. Based on the predicted probability and the hard label corresponding to the sample, a first loss value is calculated using the focus loss function; based on the first loss value, the trainable parameters of the teacher model are updated using the gradient descent method.

[0011] Furthermore, the focus loss function The expression is: , in, Batch size; For the teacher model, the first in the current batch The predicted probability of each sample; It is a parameter that adjusts the weights of easy-to-classify and difficult-to-classify samples, and its value is directly proportional to the teacher model's attention to difficult-to-classify samples; The first in the batch The class weight coefficient of each sample is determined by the hard label of that sample, specifically: , in, The first in the batch Hard labels for each sample, These are preset hyperparameters used to balance the importance of categories.

[0012] Furthermore, in step S5, the specific steps for optimizing the student model include: The teacher segmentation sequence without Bernoulli trial is input into the trained and parameter-frozen teacher model. After forward computation and temperature parameter modulation of the current teacher model, the soft label probability distribution is output. The student perturbation sequence in the perturbation segmentation sequence is input into the student model, and forward calculation is performed through the embedding layer and multiple Transformer layers of the student model to obtain the corresponding log odds. By applying a sigmoid function to the log odds of the student model and modulating the temperature parameter, the predicted probability of the student model is obtained. The distillation loss is calculated based on the predicted probability and soft label probability distribution of the student model, and the student loss is calculated based on the predicted probability and hard label of the student model. The distillation loss and student loss are then combined to obtain a second loss value. Based on the second loss value, the trainable parameters of the student model are optimized using the gradient descent method.

[0013] Furthermore, the expression for the distillation loss is: , in, and Temperature parameters for the teacher model and student model, respectively. T After modulation, the calculated batch number The probability of a sample. The expression for the student loss is: , in, For the student model, the first in the batch The predicted probability of a sample. γ The focusing parameter is used to adjust the weights of easy and difficult samples. For the first i The class weight coefficients of each sample.

[0014] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0015] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) By using denoising based on speech features and dual-round data augmentation, the robustness and generalization ability of the detection model in practical applications are improved: This invention first uses regular expressions to parse the original speech, effectively eliminating non-textual noise such as garbled characters, URLs, and usernames, avoiding the model learning irrelevant features and making it more focused on the core semantics of the text; through two rounds of data augmentation, namely character-level adversarial attack and label replacement based on Bernoulli trial, the perturbation situations such as spelling errors and unknown symbols existing in reality are simulated, so that the model can be exposed to and learn how to deal with various noise interferences during the training stage, enhancing the model's immunity to noise and its ability to recognize variant texts. This enables the deployed model to maintain stable and accurate judgment when facing complex and non-standard texts in social media, significantly reducing the false positive and false negative rates, maintaining the network environment, and ensuring the speech security of social media.

[0017] (2) By incorporating the focus loss function into the knowledge distillation process, the learning bias caused by data imbalance is alleviated, and the detection performance of the model for minority classes, i.e. harmful speech, is improved: In the supervision loss of both the teacher model fine-tuning and the student model training, the focus loss function is introduced. By adjusting the weight coefficients to balance the importance of the classes, and by increasing the weight of difficult-to-classify samples, i.e. difficult samples in the minority class, in the loss function through the focus parameter, the model will not be dominated by the majority class on datasets with serious class imbalance. Instead, it will be guided to actively learn and correctly classify those few but crucial harmful speech samples. This makes the final lightweight student model inherit and maintain the high sensitivity of the teacher model to harmful speech, thereby ensuring accurate capture of harmful speech in real imbalanced scenarios while compressing the model.

[0018] (3) By constructing a phased robust distillation framework that combines soft and hard label supervision, the inference efficiency is improved while ensuring high detection accuracy, thus meeting the real-time detection requirements: The teacher model with high detection accuracy is used as the knowledge source, and then the soft labels rich in dark knowledge output by the teacher model are transferred to the lightweight student model through distillation loss. The student loss composed of focus loss is used to ensure that the student remembers the real labels. By combining soft and hard label supervision, the student model can reproduce the complex decision boundary and representation ability of the teacher model with fewer parameters. A high-precision but bulky large model is compressed into a lightweight model that maintains similar performance but is small in size and fast in computation. This enables low-latency and high-efficiency real-time detection of harmful speech on resource-constrained devices, such as edge servers or mobile terminals, thus expanding the detection scope of harmful speech on social media. Attached Figure Description

[0019] Figure 1 A flowchart illustrating the training method for the student model; Figure 2 This is a data flow diagram for the training method of the student model. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] This embodiment provides a method for detecting harmful speech based on robust knowledge distillation. The specific steps include: collecting the original speech text to be detected from the target data source; performing the same speech denoising operation as S1 in the training method of the student model on the original speech text to be detected, to obtain the denoised speech to be detected; using the same word segmenter as in the training method of the student model, segmenting the denoised speech to be detected to generate a labeled sequence to be detected; inputting the labeled sequence to be detected into the trained student model, performing forward computation through the embedding layer and multiple Transformer layers to obtain the classification logic value of the speech; applying a sigmoid function to the classification logic value to convert it into a probability value, and outputting the final detection result of whether the speech is harmful speech based on a preset probability threshold.

[0022] The harmful speech detection method provided in this embodiment is based on a student model obtained through robust knowledge distillation, achieving accurate and robust results while consuming fewer device resources. In this application, addressing the problem of harmful speech detection being susceptible to interference from noisy samples in existing technologies, this application employs a denoising technique based on speech features. Regular expressions are used to identify and remove noisy samples, effectively solving the shortcomings of low model learning efficiency and impaired detection performance. Subsequently, the teacher model is fine-tuned on the denoised speech to ensure the effectiveness of the learned knowledge. The lightweight student model learns the knowledge from the teacher model, thereby compressing the teacher model. To mitigate bias caused by data imbalance, focus loss is incorporated into knowledge distillation to enhance the model's anti-interference ability. Furthermore, a two-round data augmentation strategy is used to enhance the model's robustness. This strategy simulates common spelling errors and unknown symbols found in the real world. The first round performs character-level adversarial attacks on the denoised speech; the second round perturbs the labeled sequence based on the results of multiple Bernoulli trials. To address the issue of knowledge distillation in existing technologies being susceptible to data imbalance and leading to learning bias in student models, this application employs a technique that integrates a focus loss function into knowledge distillation. By assigning higher weights to difficult-to-learn samples during the distillation process, the learning bias caused by data imbalance can be effectively mitigated. Furthermore, to address the insufficient robustness of existing detection models when faced with spelling errors and unknown symbols, this application utilizes a dual-round data augmentation technique. By simulating common spelling perturbations and abnormal symbol inputs, the diversity of training data is increased, effectively overcoming the poor adaptability of existing technologies.

[0023] like Figure 1 As shown, the specific steps of the student model training method in this embodiment include: S1. Collect raw speech data, and through text parsing operations based on regular expressions, identify and remove unnecessary or redundant noise elements in the raw speech data to obtain denoised speech data. S2. Randomly select a portion of the denoised speech data for character-level adversarial attacks to generate perturbation speech data, and merge the perturbation speech data with the denoised speech data to form an enhanced speech dataset; S3. Use the word segmenters corresponding to the teacher model and the student model to segment the speech in the augmented speech dataset to obtain the teacher segmentation sequence and the student segmentation sequence respectively; based on the preset perturbation probability, randomly replace the markers in the teacher segmentation sequence and the student segmentation sequence through multiple Bernoulli trials to generate the perturbation segmentation sequence; S4. Input the teacher perturbation sequence from the perturbation segmentation sequence into the teacher model, calculate the first loss value between the model prediction and the hard label based on the focus loss function, and optimize the teacher model parameters. S5. Input the student perturbation sequence from the perturbation segmentation sequence into the student model. Calculate the second loss value based on the predicted probability of the student model, the hard label, and the soft label output by the teacher model using the combined loss function. Optimize the student model parameters to obtain the trained student model. The combined loss function includes a student loss term based on the focus loss function and a distillation loss term based on the soft label of the teacher model.

[0024] like Figure 2 The diagram shows the data flow of the student model's training method. Phase one, consisting of data processing steps S1, S2, and S3, involves parsing the original speech to remove noise, followed by two rounds of data augmentation to improve the model's robustness: 1) performing character-level adversarial attacks on randomly selected samples from the denoised speech; 2) replacing the labels in the labeled sequence with unknown labels based on the results of multiple Bernoulli trials. Phase two, consisting of data processing step S4, uses a focus loss function to fine-tune the teacher model on the labeled sequence and its corresponding hard labels. Phase three, consisting of data processing step S5, freezes the teacher model and extracts soft labels for the student model to learn.

[0025] Social media posts possess inherent linguistic features, including text, gibberish, usernames, hashtags, URLs, special symbols, and repeated characters. Humans retrieve information from posts at the word level, treating URLs as units, thus easily distinguishing between key information (text) and secondary information (non-textual elements). However, most language models retrieve information at the sub-word level. Originally complete non-textual elements are broken down into multiple parts after being processed by a tokenizer, generating additional features that severely interfere with the model's extraction of key information. Therefore, in S1, the text parsing operations used for denoising include gibberish removal, special symbol removal, URL removal, repeated character elimination, username removal, and hashtag removal.

[0026] The first round of data augmentation for denoised speech data is performed using character-level adversarial attacks. The specific steps of the character-level adversarial attack in S2 include: Based on the preset perturbation ratio and the amount of denoised speech data, determine the number of adversarial attack rounds, i.e. the number of perturbation samples to be generated; From the denoised speech data, each round of attack randomly selects a speech sample using a uniform distribution; for each selected speech sample, an adversarial attacker performs character-level adversarial attack operations with a preset perturbation probability, including random character replacement, deletion, insertion, and swapping of adjacent characters, to generate perturbed speech samples; After completing all rounds of adversarial attacks, the perturbed speech samples generated in each round are combined to obtain perturbed speech data.

[0027] In S3, a second round of data augmentation is performed on the segmented sequences. The specific steps involved in randomly replacing the teacher's and student's segmented sequences through multiple Bernoulli trials include: For each token in each teacher's word segmentation sequence and student's word segmentation sequence, perform a Bernoulli trial; if the Bernoulli trial fails, replace the current token with a predefined unknown token; if the Bernoulli trial succeeds, keep the current token unchanged. After traversing the entire teacher segmentation sequence and student segmentation sequence, the corresponding perturbed segmentation sequence is obtained.

[0028] In S4, the teacher model is fine-tuned under hard-label supervision. The specific steps for optimizing the teacher model parameters include: The teacher's perturbation sequence from the perturbation segmentation sequence is input into the teacher model, and forward computation is performed through an embedding layer and multiple Transformer layers to obtain the corresponding log odds. The expression is: , in, For the teacher model, As a scalar, For the teacher perturbation sequence.

[0029] The log-odds ratio is normalized using a sigmoid function to obtain the teacher model's predicted probability for the current batch of samples. The expression is: , Obtain the probability distribution of the teacher model The model considers categories with probability values ​​greater than a preset threshold during the testing phase as the model's prediction results. This is achieved through the first probability... This represents the relationship between predicted probabilities and hard labels: .

[0030] Based on the predicted probability and the hard label corresponding to the sample, the first loss value is calculated using the focus loss function; based on the first loss value, the trainable parameters of the teacher model are updated using the gradient descent method.

[0031] Focus loss function The expression is: , in, Batch size; For the teacher model, the first in the current batch The predicted probability of each sample; It is a parameter that adjusts the weights of easy-to-classify and difficult-to-classify samples, and its value is directly proportional to the teacher model's attention to difficult-to-classify samples; The first in the batch The class weight coefficient of each sample is determined by the hard label of that sample, specifically: , in, The first in the batch Hard labels for each sample, These are preset hyperparameters used to balance the importance of categories.

[0032] In S5, the specific steps for optimizing the student model include: The teacher segmentation sequence without Bernoulli trial is input into the trained and parameter-frozen teacher model. After forward computation and temperature parameter modulation of the current teacher model, the soft label probability distribution is output. The student perturbation sequence from the perturbation segmentation sequence is input into the student model. The student model then performs forward computation through its embedding layer and multiple Transformer layers to obtain the corresponding log odds. The expression is: , in, For student models, As a scalar, For the student perturbation sequence.

[0033] By applying a sigmoid function to the log-odds of the student model and modulating it with temperature parameters, the predicted probability of the student model is obtained. : ; in Therefore, the probability distribution of the student model is: Through the second probability This represents the relationship between predicted probabilities and hard labels: .

[0034] Distillation loss is calculated based on the predicted probability of the student model and the probability distribution of the soft label. Student loss is calculated based on the predicted probability of the student model and the hard label. Distillation loss and student loss are combined to obtain a second loss value.

[0035] The student loss is used to enable the student model to retain memory of hard labels, reducing potential erroneous reliance on the teacher model. It improves the model's performance on hard-to-classify samples by introducing a focus loss function. The expression for the student loss is: , in, For the student model, the first in the batch The predicted probability of a sample. γ The focusing parameter is used to adjust the weights of easy and difficult samples. For the first i The class weight coefficients of each sample.

[0036] After optimization using the student loss, the teacher model has learned an effective classification boundary. Knowledge is transferred from the teacher model to the student model by minimizing the distillation loss, matching the probability distribution predicted by the teacher model. To ensure the stability of the teacher model's predictions, the soft labels are frozen before distillation, and a second round of data augmentation is prohibited for teacher-segmented sequences that have not undergone Bernoulli trials.

[0037] The expression for distillation loss is: , in, and Temperature parameters for the teacher model and student model, respectively. T After modulation, the calculated batch number The probability of a sample. Given the inherent semantic ambiguity of certain harmful statements, teacher models often achieve very similar confidence scores for harmful and non-harmful statements on these samples, hindering student models from effectively learning hidden knowledge. A temperature parameter is introduced to adjust the smoothness of the probability distribution. for: , , in Smaller temperature parameters make the probability distribution sharper, thus increasing the distinction between categories, but also relatively reducing the information carried by unpredicted categories; while larger temperature parameters have the opposite effect.

[0038] Based on the second loss value, the trainable parameters of the student model are optimized using the gradient descent method. The expression is: , in, These are the weighting coefficients used to adjust the importance of losses.

[0039] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0040] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0041] Multiple components in the device are connected to an I / O interface, including: input units such as a keyboard, mouse, etc.; output units such as various types of displays, speakers, etc.; storage units such as disks, optical disks, etc.; and communication units such as network interface cards, modems, wireless transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention may be implemented as a computer software program tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or the communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method of the present invention described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute the method of the present invention by any other suitable means (e.g., by means of firmware).

[0042] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0043] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0044] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0045] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting harmful speech based on robust knowledge distillation, characterized in that, The harmful speech detection method is based on a student model obtained through robust knowledge distillation. The specific steps of training the student model include: S1. Collect raw speech data, and through text parsing operations based on regular expressions, identify and remove unnecessary or redundant noise elements in the raw speech data to obtain denoised speech data. S2. Randomly select a portion of the denoised speech data to perform character-level adversarial attacks, generate perturbation speech data, and merge the perturbation speech data with the denoised speech data to form an enhanced speech dataset; S3. Use the word segmenters corresponding to the teacher model and the student model to segment the speech in the enhanced speech dataset to obtain the teacher segmentation sequence and the student segmentation sequence respectively; based on the preset perturbation probability, randomly replace the markers in the teacher segmentation sequence and the student segmentation sequence through multiple Bernoulli trials to generate a perturbation segmentation sequence; S4. Input the teacher perturbation sequence in the perturbation segmentation sequence into the teacher model, calculate the first loss value between the model prediction and the hard label based on the focus loss function, and optimize the teacher model parameters. S5. Input the student perturbation sequence in the perturbation segmentation sequence into the student model. Calculate the second loss value based on the predicted probability of the student model, the hard label, and the soft label output by the teacher model using a combined loss function. Optimize the student model parameters to obtain a trained student model. The combined loss function includes a student loss term based on the focus loss function and a distillation loss term based on the soft label of the teacher model.

2. The method for detecting harmful speech based on robust knowledge distillation according to claim 1, characterized in that, The text parsing operations used for noise reduction in S1 include garbled character repair, removal of special symbols, URL removal, duplicate character elimination, username removal, and topic tag removal.

3. The method for detecting harmful speech based on robust knowledge distillation according to claim 1, characterized in that, The specific steps of the character-level adversarial attack in S2 include: Based on the preset perturbation ratio and the amount of data in the denoised speech data, the number of adversarial attack rounds is determined, i.e., the number of perturbation samples to be generated. From the denoised speech data, each round of attack randomly selects a speech sample using a uniform distribution; for each selected speech sample, an adversarial attacker performs character-level adversarial attack operations with a preset perturbation probability, including random character replacement, deletion, insertion, and swapping of adjacent characters, to generate a perturbed speech sample; After completing all rounds of adversarial attacks, the perturbed speech samples generated in each round are combined to obtain the perturbed speech data.

4. The method for detecting harmful speech based on robust knowledge distillation according to claim 1, characterized in that, In S3, the specific steps for randomly replacing the teacher's and student's word segmentation sequences through multiple Bernoulli trials include: For each token in each teacher's word segmentation sequence and student's word segmentation sequence, perform a Bernoulli trial; if the Bernoulli trial fails, replace the current token with a predefined unknown token; if the Bernoulli trial succeeds, keep the current token unchanged. After traversing the entire teacher segmentation sequence and student segmentation sequence, the corresponding perturbed segmentation sequence is obtained.

5. The method for detecting harmful speech based on robust knowledge distillation according to claim 1, characterized in that, In step S4, the specific steps for optimizing the teacher model parameters include: The teacher perturbation sequence in the perturbation segmentation sequence is input into the teacher model, and forward calculation is performed through the embedding layer and multiple Transformer layers to obtain the corresponding log odds; The logarithmic probability is normalized using a sigmoid function to obtain the teacher model's predicted probability for the current batch of samples. Based on the predicted probability and the hard label corresponding to the sample, a first loss value is calculated using the focus loss function; based on the first loss value, the trainable parameters of the teacher model are updated using the gradient descent method.

6. The method for detecting harmful speech based on robust knowledge distillation according to claim 5, characterized in that, The focus loss function The expression is: ， in, Batch size; For the teacher model, the first in the current batch The predicted probability of each sample; It is a parameter that adjusts the weights of easy-to-classify and difficult-to-classify samples, and its value is directly proportional to the teacher model's attention to difficult-to-classify samples; The first in the batch The class weight coefficient of each sample is determined by the hard label of that sample, specifically: ， in, The first in the batch Hard labels for each sample, These are preset hyperparameters used to balance the importance of categories.

7. The method for detecting harmful speech based on robust knowledge distillation according to claim 1, characterized in that, In step S5, the specific steps for optimizing the student model include: The teacher segmentation sequence without Bernoulli trial is input into the trained and parameter-frozen teacher model. After forward computation and temperature parameter modulation of the current teacher model, the soft label probability distribution is output. The student perturbation sequence in the perturbation segmentation sequence is input into the student model, and forward calculation is performed through the embedding layer and multiple Transformer layers of the student model to obtain the corresponding log odds. By applying a sigmoid function to the log odds of the student model and modulating the temperature parameter, the predicted probability of the student model is obtained. The distillation loss is calculated based on the predicted probability and soft label probability distribution of the student model, and the student loss is calculated based on the predicted probability and hard label of the student model. The distillation loss and student loss are then combined to obtain a second loss value. Based on the second loss value, the trainable parameters of the student model are optimized using the gradient descent method.

8. The method for detecting harmful speech based on robust knowledge distillation according to claim 7, characterized in that, The expression for the distillation loss is: ， in, and Temperature parameters for the teacher model and student model, respectively. T After modulation, the calculated batch number The probability of a sample. The expression for the student loss is: ， in, For the student model, the first in the batch The predicted probability of a sample. γ The focusing parameter is used to adjust the weights of easy and difficult samples. For the first i The class weight coefficients of each sample.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images