A method for training a violation text detection model, a violation text detection method, a device, and electronic equipment.
By training the illegal text dataset with variants and salient word substitutions, a more semantically meaningful illegal text dataset is generated, which solves the problem of insufficient ability of existing models to identify new illegal text and achieves more efficient illegal text detection.
Patent Information
- Application Number
- CN202410486052.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-22
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-04-22
AI Technical Summary
Existing illegal text detection models struggle to effectively identify newly emerging illegal texts, especially when faced with human-made abstract adversarial techniques such as puns and synonyms, resulting in insufficient model generalization ability.
By inputting the illegal text dataset into the illegal text variant model for further pre-training in the domain, a variant illegal text dataset is generated. Significant words are then calculated, replaced with mask tags, and trained to form a mature illegal text detection model.
This improves the generalization and robustness of the illegal text detection model, enabling it to better identify newly emerging illegal text.
Smart Images

Figure CN118378084B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, and more specifically, to a method for training a violation text detection model, a violation text detection method, an apparatus, and an electronic device. Background Technology
[0002] In live stream chat, some inappropriate comment categories often appear. To combat these attacks, numerous manually defined rules, machine learning models, and deep learning models have been applied in practice, achieving some success. However, these methods are insufficient to resist human-induced abstraction and countermeasures. Malicious variations in text are constantly emerging, such as using puns and synonyms to deceive computers. Relying solely on data leaked from illegal industries as training data for models suffers from insufficient data volume and uneven data distribution, resulting in poor model generalization ability and an inability to effectively identify newly emerging inappropriate text. Summary of the Invention
[0003] In view of this, the purpose of this invention is to provide a method for training a violation text detection model, a method for detecting violation text, an apparatus, and an electronic device to at least partially improve the above-mentioned problems.
[0004] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, embodiments of the present invention provide a method for training a violation text detection model, the method comprising: The first dataset of violating texts is input into the violating text variant model to obtain the variant second dataset of violating texts; the first dataset of violating texts includes at least one first violating text, and the second dataset of violating texts includes at least one first violating text and at least two second violating texts, the second violating texts being obtained from the variants of the first violating text; The second illegal text dataset is input into the illegal text detection model to calculate the significant words of each of the first illegal texts; After replacing the significant words of each first illegal text in the first illegal text dataset with mask tags, the data is input into the illegal text variant model to obtain the third illegal text dataset. The third dataset of illegal texts is input into the illegal text detection model for training, resulting in a mature illegal text detection model.
[0005] Optionally, the step of inputting the obtained first violation text dataset into the violation text variant model to obtain the variant second violation text dataset includes: Obtain any one of the first violation texts; The first violating text is segmented into words to obtain at least two words; After replacing each word of the first violating text with a mask marker, the result is input into the violating text variant model to obtain at least two second violating texts corresponding to the first violating text. Iterate through each of the first violation texts to obtain the second violation text corresponding to each of the first violation texts; Merge all the first-violation texts with all the second-violation texts to obtain the second-violation text dataset.
[0006] Optionally, the step of inputting the second illegal text dataset into the illegal text detection model to calculate the significant words of each of the first illegal texts includes: Get any one of the first violating texts, and find all the second violating texts corresponding to the first violating text; The first illegal text and each of the second illegal texts are input into the illegal text detection model to obtain a first output threshold for the first illegal text and a second output threshold for each of the second illegal texts; Subtract the first output threshold from each second output threshold to obtain the output threshold difference for each second violating text. Select the second violating text with the larger output threshold difference to determine the significant words of the first violating text. Iterate through each of the first violating texts to obtain the significant words of each first violating text.
[0007] Optionally, the method further includes: The first dataset of violating texts is input into the violating text variant model, and the violating text variant model is further pre-trained in the domain.
[0008] Optionally, the step of inputting the first violating text dataset into the violating text variant model and performing further domain pre-training on the violating text variant model includes: The first illegal text dataset is segmented to obtain a fourth illegal text dataset; the fourth illegal text dataset includes at least one lexical unit. The vocabulary units are modified using a masking strategy to obtain the modified fifth violation text dataset; The violating text variant model was further pre-trained in the domain using the fifth violating text dataset.
[0009] Optionally, the violating text variant model includes a vocabulary, and the step of modifying the vocabulary units using a masking strategy to obtain the modified fifth violating text dataset includes: The masking strategy is used to modify a first preset proportion of the vocabulary units; the masking strategy is to replace a second preset proportion of the vocabulary units with mask markers and replace a third preset proportion of the vocabulary units with a random word from the vocabulary list.
[0010] Secondly, embodiments of the present invention provide a method for detecting illegal text, applied to an illegal text detection model, the method comprising: The first text to be detected is preprocessed to obtain the second text to be detected after processing. The second text to be detected is input into the illegal text detection model to obtain the output value of the illegal text detection model; When the output value is greater than a preset threshold, the first text to be detected is determined to be illegal text.
[0011] Thirdly, embodiments of the present invention provide a training apparatus for a violation text detection model, the apparatus comprising: A text variant unit is used to input the acquired first illegal text dataset into the illegal text variant model to obtain a variant second illegal text dataset; the first illegal text dataset includes at least one first illegal text, and the second illegal text dataset includes at least one first illegal text and at least two second illegal texts, wherein the second illegal text is obtained by variant of the first illegal text; The salient word search unit is used to input the second illegal text dataset into the illegal text detection model and calculate the salient words for each of the first illegal texts; The text enhancement unit is used to replace the salient words of each of the first illegal texts in the first illegal text dataset with mask tags, and then input them into the illegal text variant model to obtain the third illegal text dataset; The training unit is used to input the third illegal text dataset into the illegal text detection model for training, so as to obtain a mature illegal text detection model.
[0012] Fourthly, embodiments of the present invention provide a violation text detection device, applied to a violation text detection model, the device comprising: The text preprocessing unit is used to preprocess the acquired first text to be detected to obtain the processed second text to be detected. The illegal text detection unit is used to input the second text to be detected into the illegal text detection model and obtain the output value of the illegal text detection model. The illegal text determination unit is used to determine that the first text to be detected is illegal text when the output value is greater than a preset threshold.
[0013] Fifthly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the preceding embodiments.
[0014] This invention provides a method, apparatus, and electronic device for training a violation text detection model. The method involves inputting a first violation text dataset into a violation text variant model for further pre-training in the domain. Then, the first violation text dataset is input into the variant model again to obtain a variant second violation text dataset. This second dataset is then input into a violation text detection model to calculate the significant words of each first violation text. These significant words are then replaced with mask tags and input into the variant model again to obtain a third violation text dataset. This third dataset is then input into the violation text detection model for training, resulting in a mature violation text detection model. This allows the violation text variant model to possess better generalization and robustness, enabling it to mutate the original violation text dataset into more violation text datasets that conform to the semantics of violation text, thus allowing the violation text detection model to effectively identify newly emerging violation text.
[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 A schematic structural block diagram of an electronic device provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a method for training a violation text detection model according to an embodiment of the present invention; Figure 3 This is another flowchart illustrating a method for training a violation text detection model according to an embodiment of the present invention; Figure 4 This is another flowchart illustrating a method for training a violation text detection model according to an embodiment of the present invention; Figure 5 This is another flowchart illustrating a method for training a violation text detection model according to an embodiment of the present invention; Figure 6 This is a flowchart illustrating a method for detecting illegal text provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of a training device for a violation text detection model provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of a violation text detection device provided in an embodiment of the present invention.
[0018] Icons: 100 - Electronic device; 101 - Memory; 102 - Communication interface; 103 - Processor; 104 - Bus; 400 - Illegal text detection model training device; 410 - Text variant unit; 420 - Significant word search unit; 430 - Text enhancement unit; 440 - Training unit; 500 - Illegal text detection device; 510 - Text preprocessing unit; 520 - Illegal text detection unit; 530 - Illegal text judgment unit. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0020] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0021] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0023] With the rise and development of live streaming platforms, more and more users are watching their favorite streamers and interacting with them through comments and private messages. As the volume of user comments increases, some inappropriate categories often emerge, such as illegal advertisements, product ads, and abusive language. To combat these attacks, many manually defined rules, machine learning models, and deep learning models have been implemented with some success. However, these methods are insufficient to withstand human-induced abstract countermeasures, and malicious variations in text are constantly emerging, such as using puns and synonyms to deceive computers.
[0024] These variations include adding other irrelevant text, replacing words with homophones, similar-looking words, or synonyms, in an attempt to circumvent manual rules and text classification models. See Table 1 below for some variations of the text.
[0025] Table 1 - Text Variants Table
[0026] Therefore, relying solely on data leaked from illegal industries as training data for models can lead to problems such as insufficient data volume and uneven data distribution, resulting in insufficient generalization ability of the model and an inability to effectively identify newly emerging malicious text.
[0027] Based on the above, embodiments of the present invention provide a method for training a violation text detection model, a violation text detection method, an apparatus, and an electronic device. By inputting a first violation text dataset into a violation text variant model, a variant second violation text dataset is obtained. This second dataset is then input into the violation text detection model to calculate the significant words of each first violation text. After replacing the significant words of each first violation text dataset with mask tags, this data is input into the violation text variant model to obtain a third violation text dataset. This third dataset is then input into the violation text detection model for training, resulting in a mature violation text detection model. This allows for the generation of more violation text datasets that conform to the semantics of violation text from the original dataset, thereby enabling the violation text detection model to effectively identify newly emerging violation text.
[0028] To implement the process steps and functions of the various examples of this invention, please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, which are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.
[0029] Electronic device 100 can be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understood that electronic device 100 is not limited to a physical server, but can also be a virtual machine on a physical server, a virtual machine built on a cloud platform, or any other computer that can provide the same functionality as the server or virtual machine. The operating system of electronic device 100 can be, but is not limited to, Windows, Linux, etc.
[0030] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.
[0031] The communication connection between the electronic device 100 and external devices is achieved through at least one communication interface 102 (which can be wired or wireless).
[0032] Processor 103 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of this embodiment can be completed by integrated logic circuits in the hardware of processor 103 or by instructions in software form. Processor 103 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0033] Understandable. Figure 1 The structure shown is for illustrative purposes only; the electronic device 100 may also include components that are more advanced than those shown. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.
[0034] The following is an exemplary description of the training method for the illegal text detection model provided by this invention. Specifically, Figure 2 This is a flowchart illustrating a method for training a violation text detection model according to an embodiment of the present invention. (See attached diagram.) Figure 2 The subject executing this method can be one of the above. Figure 1 The electronic device 100 shown, the method includes as follows Figure 2The following steps are described: S210: Input the obtained first violation text dataset into the violation text variant model to obtain the variant second violation text dataset.
[0035] The first violation text dataset includes at least one first violation text, and the second violation text dataset includes at least one first violation text and at least two second violation texts, wherein the second violation texts are derived from variants of the first violation texts.
[0036] Optionally, in one possible implementation, the illegal text variant model can be obtained by pre-training the Roberta model in the domain, using a massive illegal text dataset as the training dataset. This domain-based pre-training embeds vectors of illegal text knowledge into the Roberta model, enabling the illegal text variant model to more effectively identify malicious information representations of various illegal texts commonly used in illegal industries, such as homophones, similar-looking characters, and variant texts, through contextual semantics. This gives the illegal text variant model better generalization and robustness.
[0037] S220: Input the second illegal text dataset into the illegal text detection model to calculate the significant words of each first illegal text.
[0038] Among them, prominent words indicate that the prominent word is more like a violating word in the current violating text.
[0039] Optionally, you can find one significant word or multiple significant words.
[0040] S230: Replace the salient words of each first violation text in the first violation text dataset with mask tags, and then input them into the violation text variant model to obtain the third violation text dataset.
[0041] Depending on actual needs, any ratio can be set to expand one illegal text into several illegal texts.
[0042] S240: Input the third illegal text dataset into the illegal text detection model for training to obtain a mature illegal text detection model.
[0043] The illegal text detection model can be trained based on the TextCNN model.
[0044] The training method for an illegal text detection model provided by the embodiments of the present invention inputs the acquired first illegal text dataset into the illegal text variant model to obtain a variant second illegal text dataset, inputs the second illegal text dataset into the illegal text detection model, calculates and obtains the salient words of each first illegal text, replaces the salient words of each first illegal text in the first illegal text dataset with mask tokens, inputs the result into the illegal text variant model to obtain a third illegal text dataset, and inputs the third illegal text dataset into the illegal text detection model for training to obtain a mature illegal text detection model. As a result, more illegal text datasets conforming to the semantics of illegal texts can be generated by variant processing from the original illegal text dataset, so that the illegal text detection model can effectively recognize newly emerging illegal texts.
[0045] Optionally, there are multiple ways to variant the first illegal text dataset into the second illegal text dataset, which can be performed in batches, or performed in a loop for each single text. Specifically, Figure 3 is another schematic flowchart of a training method for an illegal text detection model provided by the embodiments of the present invention, referring to Figure 3 , step S210 may specifically include the following steps: S211: Acquire any one first illegal text.
[0046] S212: Perform word segmentation processing on the first illegal text to obtain at least two words.
[0047] S213: Replace each word of the first illegal text with a mask token respectively, input the result into the illegal text variant model, and obtain at least two second illegal texts corresponding to the first illegal text.
[0048] S214: Traverse each first illegal text to obtain the second illegal text corresponding to each first illegal text.
[0049] S215: Combine all the first illegal texts and all the second illegal texts to obtain the second illegal text dataset.
[0050] One first illegal text is acquired, and the first illegal text is segmented by a word segmentation tool. The first illegal text is a phrase or a long sentence, and at least two words can be obtained after word segmentation.
[0051] For example, if the illegal text is "Welfare please click", word segmentation is performed on it to obtain three words "Welfare", "please" and "click". For these three words, they are respectively marked as masks to obtain three texts, which are "[MASK] please click", "Welfare [MASK] click" and "Welfare please [MASK]".
[0052] In step S213, the first violating text after masking can also be input into a thesaurus, and the second violating text can be obtained through the synonym variant. The thesaurus can be WordNet or HowNet, etc.
[0053] Variations can be performed on the first violating text after the mask mark, either one-to-one or one-to-many.
[0054] If a one-to-one variation is performed, in this example, the second violation text may include "Fuli please click", "welfare light click", and "welfare please electrode", which can then be merged with the first violation text to obtain the second violation text dataset [welfare please click, Fuli please click, welfare light click, welfare please electrode].
[0055] If we make a variation of the 1:2 ratio, in this example, we might get a second violation text including "R&F please click", "Compound interest please click", "Welfare light click", "Welfare green click", "Welfare please electrode", "Welfare please motor", which, after merging, we get [Welfare please click, Compound interest please click, R&F please click, Welfare green click, Welfare light click, Welfare please electrode, Welfare please motor].
[0056] For the words "benefits," "please," and "click," we need to find words that are more likely to be in violation. One possible implementation method is... Figure 4 This is another flowchart illustrating a method for training a violation text detection model according to an embodiment of the present invention. See also... Figure 4 Step S220 may specifically include the following steps: S221: Get any first violation text and find all second violation texts corresponding to the first violation text.
[0057] S222: Input the first illegal text and each second illegal text into the illegal text detection model to obtain the first output threshold of the first illegal text and the second output threshold of each second illegal text.
[0058] The output threshold represents the probability that the text is illegal.
[0059] S223: Subtract the first output threshold from each second output threshold to obtain the output threshold difference for each second violating text. Select the second violating text with the larger output threshold difference to determine the significant words of the first violating text.
[0060] Repeat steps S221 to S223 until the significant words of each first violation text are obtained.
[0061] In the example above, find the first violating text "Click for benefits" and the corresponding second violating text.
[0062] If the variation is performed on a one-to-one basis, the second non-compliant texts are "Fuli Please Click", "Benefit Light Click", and "Benefit Please Electrode". The first non-compliant text and the second non-compliant texts are input into a non-compliant text detection model to obtain their corresponding output thresholds respectively, and then the difference between the output thresholds of the second non-compliant texts is calculated. For example, the output threshold of the first non-compliant text "Benefit Please Click" is 90%, as shown in Table 2.
[0063] Table 2 - First Output Threshold Comparison Table
[0064] Words with large output threshold differences are taken as salient words. If one salient word is selected, "benefit" is taken as the salient word; if two salient words are selected, "benefit" and "click" are taken as the salient words. It is also possible to determine whether to select a word as a salient word according to whether the output threshold difference exceeds a preset value. For example, if the preset value is 50%, there is only one salient word, which is "benefit".
[0065] If the variation is performed on a one-to-two basis, the second non-compliant texts are "Fuli Please Click", "Compound Interest Please Click", "Benefit Light Click", "Benefit Green Click", "Benefit Please Electrode", and "Benefit Please Motor". For example, the output threshold of the first non-compliant text "Benefit Please Click" is 90%, as shown in Table 2.
[0066] Table 3 - Second Output Threshold Comparison Table
[0067] In the case of one-to-many variation, the average value of the output threshold differences is calculated, and then words with a larger average value are selected as salient words. If one salient word is selected, "benefit" is taken as the salient word; if two salient words are selected, "benefit" and "click" are taken as the salient words.
[0068] In this example, in step S230, the salient word of each first non-compliant text in the first non-compliant text dataset is replaced with a mask token. Any ratio can be set to expand one non-compliant text into several non-compliant texts.
[0069] If one salient word is selected and the salient word is "benefit", the text obtained after masking the salient word of "Benefit Please Click" is "[MASK] Please Click". If the first non-compliant text is expanded at a ratio of 1:3, a third non-compliant text dataset of "Fuli Please Click", "Compound Interest Please Click", and "Buoyancy Please Click" may be obtained.
[0070] If two prominent words are selected, namely "welfare" and "click", then the text after masking the prominent words of "welfare please click" will be "[MASK] please click" and "welfare please [MASK]". If the first violating text is amplified at a ratio of 1:3, a third violating text dataset may be obtained, which includes "Fuli please click", "compound interest please click", "buoyancy please click", "welfare please remember", "welfare please classics", and "welfare please electric shock".
[0071] The amplified third dataset of illegal texts was input into the illegal text detection model for training, resulting in a mature illegal text detection model. This third dataset of illegal texts has been amplified and subjected to salient word search, making it more semantically relevant to illegal industries. Therefore, the illegal text detection model trained with this third dataset can improve its ability to identify newly emerging variants of malicious texts.
[0072] To improve the generalization and robustness of the violating text variant model and to generate more violating text datasets that conform to the semantics of violating texts from the original violating text dataset, the violating text variant model can be pre-trained in the domain using a massive first violating text dataset. Specifically, Figure 5 This is another flowchart illustrating a method for training a violation text detection model according to an embodiment of the present invention. See also... Figure 5 The training method for this illegal text detection model may also include the following steps: S220: Input the first illegal text dataset into the illegal text variant model, and perform further pre-training of the illegal text variant model in the domain.
[0073] The specific steps of step S220 may include: The first illegal text dataset is segmented to obtain the fourth illegal text dataset; the fourth illegal text dataset includes at least one lexical unit.
[0074] The fifth violation text dataset was obtained by modifying the vocabulary units using a masking strategy.
[0075] The domain-specific pre-training of the variant model of the variant text was performed using the fifth variant text dataset.
[0076] The violating text variant model includes a vocabulary, and the steps to obtain the modified fifth violating text dataset by modifying the vocabulary units using a masking strategy include: A masking strategy is used to modify a first preset proportion of vocabulary units. The masking strategy involves replacing a second preset proportion of vocabulary units with mask markers and replacing a third preset proportion of vocabulary units with a random word from the vocabulary list.
[0077] In the preferred scheme, the first preset ratio is 15%, the second preset ratio is 80%, and the third preset ratio is 10%. That is, 85% of the tokens (vocabulary units) are not modified by the masking strategy, and the remaining 15% of the tokens are modified by the masking strategy. The specific masking strategy is as follows: replace with the [MASK] tag with an 80% probability; replace with any random word in the vocabulary with a 10% probability; and keep the original word unchanged with a 10% probability, i.e., do not replace it.
[0078] After training the aforementioned illegal text detection model, a mature illegal text detection model is obtained. This model can be used to detect text to be detected. This embodiment of the invention provides an illegal text detection method applied to the aforementioned illegal text detection model. Figure 6 This is a flowchart illustrating a method for detecting illegal text provided in an embodiment of the present invention. See [link / reference]. Figure 6 The method for detecting illegal text includes: Step S310: Preprocess the first text to be detected to obtain the processed second text to be detected.
[0079] Step S320: Input the second text to be detected into the illegal text detection model to obtain the output value of the illegal text detection model.
[0080] Step S330: When the output value is greater than the preset threshold, the first text to be detected is determined to be illegal text.
[0081] If a user submits a comment, after obtaining the comment, the input-output sequence of the comment is first processed to meet the input requirements of the violation text detection model. The comment is then input into the violation text detection model to obtain the output value of the violation text detection model, for example, 60%. If the preset threshold is 70%, the comment is determined not to be violation text; if the preset threshold is 50%, the comment is determined to be violation text.
[0082] In one embodiment, the variant ratio can be 1:1, only one significant word is searched, and the data is amplified at a ratio of 1:3. After using this violation text detection method, the results are shown in Tables 4 and 5.
[0083] Table 4 - Performance of the Illegal Text Detection Model
[0084] Table 5 - Effectiveness of Illegal Text Detection Methods
[0085] Based on the above example, a training device 400 for a violation text detection model is provided below. Specifically, Figure 7This is a schematic diagram of a training device for a violation text detection model provided in an embodiment of the present invention. See also: Figure 7 The device includes: a text variant unit 410, a salient word search unit 420, a text enhancement unit 430, and a training unit 440. The text variant unit 410 is used to input the acquired first illegal text dataset into the illegal text variant model to obtain the variant second illegal text dataset; the first illegal text dataset includes at least one first illegal text, and the second illegal text dataset includes at least one first illegal text and at least two second illegal texts, and the second illegal text is obtained by the variant of the first illegal text.
[0086] The salient word search unit 420 is used to input the second illegal text dataset into the illegal text detection model and calculate the salient words for each first illegal text.
[0087] The text enhancement unit 430 is used to replace the salient words of each first violating text in the first violating text dataset with mask tags, and then input them into the violating text variant model to obtain the third violating text dataset.
[0088] Training unit 440 is used to input the third illegal text dataset into the illegal text detection model for training, so as to obtain a mature illegal text detection model.
[0089] Furthermore, the present invention also provides a non-compliant text detection device 500. Specifically, Figure 8 This is a schematic diagram of a violation text detection device provided in an embodiment of the present invention. See also: Figure 8 The illegal text detection device includes: a text preprocessing unit 510, an illegal text detection unit 520, and an illegal text determination unit 530. The text preprocessing unit 510 is used to preprocess the acquired first text to be detected to obtain the processed second text to be detected.
[0090] The illegal text detection unit 520 is used to input the second text to be detected into the illegal text detection model to obtain the output value of the illegal text detection model.
[0091] The illegal text determination unit 530 is used to determine that the first text to be detected is illegal text when the output value is greater than a preset threshold.
[0092] In summary, the illegal text detection model training method, illegal text detection method, device, and electronic device provided by this invention involve inputting a first illegal text dataset into an illegal text variant model for further domain-specific pre-training. Then, the acquired first illegal text dataset is input into the illegal text variant model to obtain a variant second illegal text dataset. This second dataset is then input into the illegal text detection model to calculate the significant words of each first illegal text. After replacing the significant words of each first illegal text dataset with mask tags, this is input into the illegal text variant model to obtain a third illegal text dataset. This third dataset is then input into the illegal text detection model for training, resulting in a mature illegal text detection model. This allows the illegal text variant model to possess better generalization and robustness, enabling it to mutate the original illegal text dataset into more illegal text datasets that conform to the semantics of illegal text, thus allowing the illegal text detection model to effectively identify newly emerging illegal text.
[0093] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0094] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0095] If the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0096] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0097] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A method for training a violation text detection model, characterized in that, The method includes: The process of inputting the acquired first violation text dataset into a violation text variant model to obtain a variant second violation text dataset includes: acquiring any first violation text; performing word segmentation on the first violation text to obtain at least two words; replacing each word of the first violation text with a mask tag and inputting it into the violation text variant model to obtain at least two second violation texts corresponding to the first violation text; traversing each first violation text to obtain the second violation text corresponding to each first violation text; and merging all the first violation texts with all the second violation texts to obtain the second violation text dataset. The first violation text dataset includes at least one first violation text, and the second text dataset includes at least one first violation text and at least two second violation texts, wherein the second violation texts are obtained from the variant of the first violation text. The second illegal text dataset is input into the illegal text detection model to calculate the significant words of each first illegal text, including: obtaining any first illegal text and finding all second illegal texts corresponding to the first illegal text; inputting the first illegal text and each second illegal text into the illegal text detection model to obtain a first output threshold for the first illegal text and a second output threshold for each second illegal text; subtracting the first output threshold from each second output threshold to obtain the output threshold difference for each second illegal text, selecting the second illegal text with the larger output threshold difference, and determining the significant words of the first illegal text; traversing each first illegal text to obtain the significant words of each first illegal text. After replacing the significant words of each first illegal text in the first illegal text dataset with mask tags, the data is input into the illegal text variant model to obtain the third illegal text dataset. The third dataset of illegal texts is input into the illegal text detection model for training, resulting in a mature illegal text detection model.
2. The method of claim 1, wherein, The method further includes: The first dataset of violating texts is input into the violating text variant model, and the violating text variant model is further pre-trained in the domain.
3. The method of claim 2, wherein, The step of inputting the first illegal text dataset into the illegal text variant model and performing further domain pre-training on the illegal text variant model includes: The first illegal text dataset is segmented to obtain a fourth illegal text dataset; the fourth illegal text dataset includes at least one lexical unit. The vocabulary units are modified using a masking strategy to obtain the modified fifth violation text dataset; The violating text variant model was further pre-trained in the domain using the fifth violating text dataset.
4. The method of claim 3, wherein, The violating text variant model includes a vocabulary, and the step of modifying the vocabulary units using a masking strategy to obtain the modified fifth violating text dataset includes: The masking strategy is used to modify a first preset proportion of the vocabulary units; the masking strategy is to replace a second preset proportion of the vocabulary units with mask markers and replace a third preset proportion of the vocabulary units with a random word from the vocabulary list.
5. A method of detecting a violation text, characterized by, The method is applied to a violation text detection model, wherein the violation text detection model is trained by the violation text detection model training method as described in any one of claims 1 to 4, the method comprising: The first text to be detected is preprocessed to obtain the second text to be detected after processing. The second text to be detected is input into the illegal text detection model to obtain the output value of the illegal text detection model; When the output value is greater than a preset threshold, the first text to be detected is determined to be illegal text. 6.A device for training a violation text detection model, characterized in that, The device includes: A text variant unit is used to input the acquired first violation text dataset into a violation text variant model to obtain a variant second violation text dataset. The process includes: acquiring any first violation text; performing word segmentation on the first violation text to obtain at least two words; replacing each word of the first violation text with a mask marker and inputting the result into the violation text variant model to obtain at least two second violation texts corresponding to the first violation text; traversing each first violation text to obtain the second violation text corresponding to each first violation text; and merging all first violation texts with all second violation texts to obtain the second violation text dataset. The first violation text dataset includes at least one first violation text, and the second text dataset includes at least one first violation text and at least two second violation texts, wherein the second violation texts are obtained from the variant of the first violation text. The salient word search unit is used to input the second illegal text dataset into the illegal text detection model and calculate the salient words of each first illegal text. This includes: obtaining any first illegal text and finding all second illegal texts corresponding to the first illegal text; inputting the first illegal text and each second illegal text into the illegal text detection model to obtain a first output threshold for the first illegal text and a second output threshold for each second illegal text; subtracting the first output threshold from each second output threshold to obtain the output threshold difference for each second illegal text; selecting the second illegal text with the larger output threshold difference to determine the salient words of the first illegal text; and traversing each first illegal text to obtain the salient words of each first illegal text. The text enhancement unit is used to replace the salient words of each of the first illegal texts in the first illegal text dataset with mask tags, and then input them into the illegal text variant model to obtain the third illegal text dataset; The training unit is used to input the third illegal text dataset into the illegal text detection model for training, so as to obtain a mature illegal text detection model.
7. A device for detecting illegal text, characterized in that, An apparatus for use in a violation text detection model, wherein the violation text detection model is trained by the violation text detection model training method as described in any one of claims 1 to 4, the apparatus comprising: The text preprocessing unit is used to preprocess the acquired first text to be detected to obtain the processed second text to be detected. The illegal text detection unit is used to input the second text to be detected into the illegal text detection model and obtain the output value of the illegal text detection model. The illegal text determination unit is used to determine that the first text to be detected is illegal text when the output value is greater than a preset threshold.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Method and device for detecting financial transaction communication text and electronic equipment
CN116244407A
