Generalized keyword-based text detection method and apparatus, and electronic device
By splitting keywords into word tuples and filtering based on frequency and occurrence count, the problem of missed detections in keyword pre-filtering mechanisms under resource-constrained environments is solved, achieving efficient identification of malicious text and reducing false positive rate.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING HILLSTONE NETWORKS INFORMATION TECHCO
- Filing Date
- 2025-12-17
- Publication Date
- 2026-04-24
AI Technical Summary
In resource-constrained environments, existing keyword pre-filtering mechanisms are too simplistic and prone to missing malicious text, with a high false positive rate, and cannot effectively identify word variations and abbreviations.
Keywords are split into multiple word tuples. Target word tuples are selected based on their frequency in positive sample texts. The number of times the target word tuples appear in the target text is used to make a judgment. Lightweight decision logic is used for text screening.
It improves the ability to recognize word variants and abbreviations, reduces the false positive rate, meets the needs of efficient detection in resource-constrained environments, and reduces computational complexity.
Smart Images
Figure CN121920355A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of information security and artificial intelligence, and more specifically, to a text detection method, apparatus, and electronic device based on generalized keywords. Background Technology
[0002] With the widespread application of large language models, content security risks are becoming increasingly prominent, making efficient detection of malicious prompts and inappropriate content a critical requirement. In resource-constrained environments, such as edge computing devices or local servers equipped only with a central processing unit (CPU), it is necessary to balance detection performance and accuracy under limited computing power.
[0003] Among related technologies, text content security detection mainly relies on keyword matching, embedding models, and large language models. Keyword matching is the fastest, but it has a high false negative and false positive rate. Large language models have high accuracy, but are slow and computationally expensive. Embedding models achieve a certain balance between performance and accuracy, but their direct application in high-concurrency or resource-constrained scenarios still faces challenges. Common pre-filtering schemes are mostly based on direct keyword matching.
[0004] However, existing keyword-based pre-filtering mechanisms have significant technical shortcomings. These methods typically rely on matching the complete string of predefined keywords, lacking the ability to recognize variations, abbreviations, neologisms, or partially matched text. In resource-constrained environments, they are prone to missed detections due to overly simplistic rules. Furthermore, by not considering the commonality of keyword fragments in normal contexts and directly using complete keywords or their common variations as matching rules, they are prone to generating false positives in harmless text, potentially unnecessarily consuming subsequent computational resources.
[0005] There is currently no effective solution to the above problems. Summary of the Invention
[0006] This application provides a text detection method, apparatus, and electronic device based on generalized keywords, which at least solves the technical problem in the prior art that the keyword pre-filtering mechanism is too simple and easily misses malicious text in resource-constrained environments.
[0007] According to one aspect of the embodiments of this application, a text detection method based on generalized keywords is provided, comprising: obtaining a keyword set; splitting each keyword in the keyword set into multiple word tuples, wherein each word tuple consists of at least one character; filtering at least one target word tuple from all the split word tuples according to the frequency of occurrence of each word tuple in positive sample text, wherein the frequency of occurrence of the target word tuple in the positive sample text is less than a preset frequency, the preset frequency corresponding to different types of target word tuples is different, and the preset frequency corresponding to each type of target word tuple is determined based on the statistical results of the number of times each type of target word tuple appears in the sample; and determining whether the target text is suspicious text based on the number of times the target word tuple appears in the target text.
[0008] Optionally, determining whether the target text is suspicious based on the number of times the target word tuple appears in the target text includes: obtaining a first threshold; and determining whether the target text is suspicious based on the number of times the target word tuple appears in the target text and the first threshold.
[0009] Optionally, obtaining the first threshold includes: counting the number of times the target word tuple appears in each positive sample text and the number of times the target word tuple appears in each negative sample text; and determining the first threshold based on the number of times the target word tuple appears in each positive sample text and the number of times the target word tuple appears in each negative sample text.
[0010] Optionally, the determination of whether the target text is suspicious text is based on the number of times the target word tuple appears in the target text and a first threshold, including: if the number of times the target word tuple appears in the target text is greater than or equal to the first threshold, the target text is determined to be suspicious text; if the number of times the target word tuple appears in the target text is less than the first threshold, the target text is determined not to be suspicious text, and the target text is filtered.
[0011] Optionally, determining whether the target text is suspicious text based on the frequency of occurrence of the target word tuple in the target text includes: determining the type of the target word tuple based on the number of characters included in the target word tuple, wherein target word tuples composed of different numbers of characters belong to different types of word tuples; determining the weight of the target word tuple based on its type; performing a weighted calculation based on the weight of the target word tuple and the frequency of occurrence of the target word tuple in the target text to obtain a weighted calculation result; and determining whether the target text is suspicious text based on the weighted calculation result.
[0012] Optionally, determining whether the target text is suspicious text based on the weighted calculation result includes: calculating the ratio of the weighted calculation result to the text length of the target text to obtain a target score; if the target score is detected to be greater than a second threshold, determining that the target text is suspicious text, wherein the second threshold is a score threshold determined using the weighted calculation method and the text length of the sample; if the target score is detected to be less than or equal to the second threshold, determining that the target text is not suspicious text, and filtering the target text.
[0013] Optionally, the weight of the target word tuple is negatively correlated with the number of characters included in the target word tuple.
[0014] Optionally, the text detection method based on generalized keywords further includes: using statistical method one or statistical method two to count the number of times the target word tuple appears in the target text, and to count the number of times each word tuple appears in the positive sample text and the negative sample text; wherein, statistical method one includes: when the same word tuple is detected to appear N times in the same text, the number of times the word tuple appears in the text is counted only once, where N is an integer greater than or equal to 1; wherein, statistical method two includes: when the same word tuple is detected to appear N times in the same text, the number of times the word tuple appears in the text is recorded as N times.
[0015] According to another aspect of the embodiments of this application, a text detection device based on generalized keywords is also provided, comprising: an acquisition unit for acquiring a keyword set; a first processing unit for splitting each keyword in the keyword set into multiple word tuples, wherein each word tuple consists of at least one character; a second processing unit for filtering at least one target word tuple from all the split word tuples according to the frequency of occurrence of each word tuple in positive sample text, wherein the frequency of occurrence of the target word tuple in the positive sample text is less than a preset frequency, the preset frequencies corresponding to different types of target word tuples are different, and the preset frequency corresponding to each type of target word tuple is determined based on the statistical results of the number of times each type of target word tuple appears in the sample; and a third processing unit for determining whether the target text is suspicious text according to the number of times the target word tuple appears in the target text.
[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the above-described text detection method based on generalized keywords.
[0017] In this application, the content security review system first obtains a keyword set and splits each keyword in the keyword set into multiple word tuples, where each word tuple consists of at least one character. Then, based on the frequency of each word tuple in the positive sample text, at least one target word tuple is selected from all the split word tuples, wherein the frequency of the target word tuple in the positive sample text is less than a preset frequency. Finally, based on the number of times the target word tuple appears in the target text, it is determined whether the target text is suspicious text.
[0018] As described above, the content security review system first transforms keyword matching from full-word matching to fragment matching by splitting keywords into word tuples. This allows the detection to capture character fragments in the text that are similar to risky keywords, thereby enhancing the ability to identify word variations, abbreviations, and partial content, and directly increasing coverage to reduce missed detections. Secondly, filtering target word tuples based on the frequency of positive sample text filters out common interference fragments in normal text, retaining more distinctive features and helping to reduce false positives caused by matching harmless common combinations, making pre-filtering judgments more accurate. Finally, threshold judgment based on the occurrence frequency of target word tuples constitutes a lightweight decision logic that only requires counting and comparison, ensuring that the entire process can be executed efficiently in an environment equipped only with a central processing unit, meeting the stringent speed requirements of the pre-filtering stage. This application, without excessively increasing computational complexity, effectively improves the detection capability of potentially malicious text under limited computing power by refining feature granularity and refining feature quality, thus solving the technical problem in existing technologies where keyword pre-filtering mechanisms are too simple and prone to missing malicious text in resource-constrained environments. Attached Figure Description
[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0020] Figure 1 This is a schematic diagram of an optional text detection method based on generalized keywords according to an embodiment of this application;
[0021] Figure 2 This is a schematic diagram illustrating an optional process for obtaining a first threshold according to an embodiment of this application;
[0022] Figure 3 This is a schematic diagram of another optional first threshold acquisition process according to an embodiment of this application;
[0023] Figure 4 This is a schematic diagram of an optional text detection device based on generalized keywords according to an embodiment of this application. Detailed Implementation
[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0026] According to an embodiment of this application, a method embodiment of a text detection method based on generalized keywords is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0027] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.
[0028] According to the embodiments of this application, a content security audit system (hereinafter referred to as the system) can be used as the execution subject of the text detection method based on generalized keywords in the embodiments of this application. The system can be a software system or an embedded system combining software and hardware. Of course, the execution subject of the method in the embodiments of this application can also be other forms of execution subject, such as devices, equipment, etc. It should be known by those skilled in the art that this application does not particularly limit the specific form of the execution subject.
[0029] Figure 1 This is a text detection method based on generalized keywords according to embodiments of this application, such as... Figure 1 As shown, the method includes the following steps:
[0030] Step S101: Obtain the keyword set.
[0031] Optionally, obtaining a keyword set represents the operation of reading a predefined set of keywords from an external data source or internal storage unit. The keyword set can be associated with a specific risk content category. The purpose of executing step S101 is to provide a basic feature source for subsequent text analysis processing, to determine the initial operation object of the detection process, so that subsequent generalization processing has a clear input and target scope, which is conducive to improving the targeting and efficiency of the entire detection process.
[0032] Optionally, obtaining the keyword set can be achieved by loading a keyword list from a pre-defined sensitive word database. Another approach to obtaining the keyword set can be to automatically extract and summarize frequently occurring risk words to form a keyword set by analyzing historical risk event records.
[0033] Step S102: Split each keyword in the keyword set into multiple word tuples, wherein each word tuple consists of at least one character.
[0034] Optionally, a tuple is used to represent a subsequence unit consisting of at least one consecutive character. Splitting each keyword in the keyword set into multiple tuples refers to the operation of segmenting each complete keyword string according to specific character length rules to generate a series of tuples. The main purpose of executing step S102 is to overcome the limitation of exact matching of complete keyword strings, enabling the detection method to identify character fragments in the target text that are similar to the original keyword portion. This enhances the content security audit system's ability to identify lexical variations, abbreviations, or partial content matching, and helps reduce missed detections caused by the requirement for complete keyword matching.
[0035] Optionally, a word tuple can include units consisting of a single character, or it can include units consisting of two consecutive characters. For example, a keyword ABCD can be split into word tuples consisting of two characters, resulting in three word tuples: AB, BC, and CD. One way to split each keyword in a keyword set is to use a sliding truncation algorithm with a fixed window length. Another way to split each keyword in a keyword set is to use sliding truncation with multiple window lengths simultaneously to generate word tuple sets of different character lengths.
[0036] Optionally, when splitting each keyword in the keyword set into multiple tuples, each tuple can consist of at least one character; that is, the splitting operation is not limited to generating fragments composed of a fixed number of characters. For example, for a keyword ABCD (used here as an example to illustrate the principle, where A, B, C, and D represent different characters), the content security auditing system can split it into tuples consisting of only a single character, such as A, B, C, and D, or into tuples consisting of two consecutive characters, such as AB, BC, and CD. Multi-granularity splitting can enhance feature generalization ability. By introducing tuples with fewer characters, such as single-character tuples, the content security auditing system can capture risky content fragments in the target text that only partially overlap with the original keywords, thereby expanding the coverage of semantically similar words not explicitly listed in the keyword set, which helps in detecting word variations or neologisms.
[0037] Step S103: Based on the frequency of each word tuple in the positive sample text, at least one target word tuple is selected from all the split word tuples.
[0038] Among them, the frequency of the target word tuple in the positive sample text is less than the preset frequency. The preset frequencies corresponding to different types of target word tuples are different, and the preset frequency corresponding to each target word tuple is determined based on the statistical results of the number of times each target word tuple appears in the sample.
[0039] Optionally, positive sample text represents a set of texts known to be safe or normal. Occurrence frequency characterizes the statistical frequency of tuple occurrences in a large number of positive sample texts. Preset frequency is a pre-defined numerical threshold. Target tuples represent those tuples obtained after filtering whose occurrence frequency is lower than the preset frequency. Filtering based on the occurrence frequency of each tuple in the positive sample text can perform feature purification on the initial tuple set, which helps to filter out common character fragments that are also prevalent in normal text, retaining those tuples that are relatively rare in normal contexts and are more likely to have risk indication significance as target tuples. This helps reduce the possibility of misjudgment due to matching common tuples in subsequent judgment processes.
[0040] Optionally, the process of selecting target word tuples may include: first, counting the frequency of each word tuple in a large-scale positive sample text set; then, comparing the frequency of each word tuple with a preset frequency; and finally, extracting all word tuples with a frequency lower than the preset frequency to form a target word tuple set. The preset frequency can be set according to the tolerance for false positives in the actual application scenario. For example, the preset frequency can be set as an absolute upper limit or as an upper limit of a relative occurrence ratio. In addition, different types of target word tuples correspond to different preset frequencies. For example, since untuples (i.e., target word tuples consisting of one character) appear more frequently in all positive samples than bigrams (i.e., target word tuples consisting of two characters), untuples and bigrams will have different preset frequencies.
[0041] Optionally, the preset frequency of each target word tuple is determined based on the statistical results of the number of times the target word tuple appears in the sample. For example, according to the statistical results, if the frequency of a tuple in different samples is 0, 3, 8, 300, 400, 500, then 50 can be taken as the preset frequency corresponding to the tuple.
[0042] Step S104: Determine whether the target text is suspicious based on the number of times the target word tuple appears in the target text.
[0043] Optionally, the target text represents the unknown text to be detected. The occurrence count represents the cumulative number of target word tuples matched in the target text. The purpose of judging based on the occurrence count of target word tuples in the target text is to make a rapid decision on the risk attributes of the target text based on the quantified matching results. This provides a classification rule with low computational complexity, enabling the entire detection method to be executed efficiently in resource-constrained environments, completing the initial screening of a large amount of text, and outputting the target text judged as suspicious for further analysis.
[0044] Alternatively, one way to determine whether a target text is suspicious is to use a first threshold. For example, the total number of occurrences of the target word tuple in the target text can be calculated; the total number of occurrences can be compared with a first threshold; if the total number of occurrences is greater than or equal to the first threshold, the target text is determined to be suspicious; if the total number of occurrences is less than the first threshold, the target text is determined not to be suspicious. The value of the first threshold can be determined based on the statistical distribution of the target word tuple on a known sample set.
[0045] This application's embodiments firstly, by splitting keywords into word tuples, transforms the matching process from complete strings to partial fragments, enhancing the ability to capture fragments in the text that are similar to but not identical to the keywords. Secondly, using positive sample text to perform frequency-based filtering of word tuples retains target word tuples with higher discriminative power, contributing to improved accuracy. Finally, the logic for judging based on the frequency of target word tuple occurrences is computationally simple and suitable for rapid execution in a central processing unit environment, meeting the speed requirements of the pre-filtering stage. The entire method, without excessively increasing the computational burden, improves the coverage of potential risk text identification through a combination of feature generalization and feature filtering.
[0046] In one optional embodiment, determining whether a target text is suspicious based on the number of times the target word tuple appears in the target text includes: the content security review system can obtain a first threshold, and then determine whether the target text is suspicious based on the number of times the target word tuple appears in the target text and the first threshold.
[0047] Optionally, the first threshold is used to represent a numerical benchmark for comparison with the frequency of occurrence. Obtaining the first threshold provides a clear quantitative standard for judging whether a target text is suspicious, transforming the judgment process from experience-based qualitative analysis to objective decision-making based on numerical comparison, which helps improve the consistency and repeatability of the judgment results. The first threshold can be a fixed empirical value or a value dynamically adjusted based on data.
[0048] Optionally, judging based on the frequency of target word tuples in the target text and a first threshold aims to execute a clear binary classification decision rule. This allows for the final output of the detection process, has simple computational logic involving only comparison operations, and is well-suited for efficient execution in a central processing unit environment, thus meeting the speed requirements of the pre-filtering stage. Specifically, the judgment process can be: comparing the total frequency of target word tuples in the target text with the first threshold, and outputting a conclusion on whether the target text is suspicious based on the comparison result.
[0049] In one optional embodiment, obtaining the first threshold includes: the content security review system can count the number of times the target word tuple appears in each positive sample text and the number of times the target word tuple appears in each negative sample text. Then, the first threshold is determined based on the number of times the target word tuple appears in each positive sample text and the number of times the target word tuple appears in each negative sample text.
[0050] Optionally, negative sample text is used to represent a set of texts known to contain risky or inappropriate content. The purpose of the content security review system in counting the occurrences of target word tuples in positive and negative sample texts is to collect distribution data of target word tuples in different categories of text. This provides a data-driven basis for determining the first threshold, enabling the setting of the first threshold to reflect the distinguishing power of the target word tuples in actual data scenarios, and helping to make the threshold setting more targeted and reasonable.
[0051] Optionally, the purpose of the content security review system in determining the first threshold based on the statistical occurrence frequency is to find an optimized classification boundary based on the actual distribution of the data. This can improve the applicability of the first threshold in distinguishing between positive and negative sample text, thereby optimizing the accuracy of subsequent judgment steps.
[0052] Optionally, the content security review system can analyze the distribution of the frequency of target word tuples in positive and negative sample texts, for example, by plotting a distribution histogram or calculating the cumulative distribution, and then select a numerical point that can balance the classification effect of the two types of samples as the first threshold. This data-driven threshold setting method can set the first threshold based on the statistically analyzed distribution of the number of specific target word tuples in positive and negative sample texts.
[0053] For example, the first threshold can be determined based on the statistical distribution of the target word tuples. The content security review system can count the number of times the target word tuple appears in each positive sample text, obtaining a series of values, such as 11, 23, 5; simultaneously, it can count the number of times the target word tuple appears in each negative sample text, obtaining another series of values, such as 123, 98, 74. By analyzing the distribution characteristics of these two sets of statistical data, such as comparing the numerical range or central tendency, the content security review system can determine a value that can effectively distinguish between the two types of samples as the first threshold, based on the number of times the target word tuple appears in each positive sample text and each negative sample text. For example, the first threshold could be set to 50.
[0054] Figure 2 A schematic diagram illustrating the process of obtaining a first threshold is shown. The content security review system breaks down the keyword set into multiple word tuples consisting of two characters, such as AB, BC, CD, etc. First, the content security review system filters based on the frequency of each word tuple in positive sample text to obtain target word tuples. Second, the content security review system determines the first threshold based on the statistical occurrence count of the target word tuples in both negative and positive sample text. For example, Figure 2The right side displays examples of the occurrence counts of the target word tuple in negative sample text (e.g., 123, 98, 74) and positive sample text (e.g., 11, 23, 5). Based on the distribution of these two sets of data—for example, a generally higher occurrence count in negative sample text and a generally lower occurrence count in positive sample text—the content security review system determines a distinguishing value as a first threshold, which can be selected as 50. Ultimately, the content security review system uses this first threshold to determine whether the target word tuple is suspicious text based on its occurrence count in the target text to be detected.
[0055] Figure 3 This diagram illustrates another process for obtaining the first threshold. The content security review system splits the keyword set into two types of target word tuples: word tuples consisting of a single character (e.g., X, Y, Z) and word tuples consisting of two characters (e.g., AB, BC, CD). These word tuples can represent multiple candidate segments split from the keywords. The content security review system then filters based on the frequency of each word tuple in positive sample text to obtain target word tuples. Next, the content security review system counts the occurrences of these target word tuples in both negative and positive sample texts. For example, 2-gram target word tuples appear 123, 98, and 74 times in negative sample text, and 11, 23, and 5 times in positive sample text. Examples of occurrences of 1-gram target word tuples in positive sample text are 12, 7, 35, and 29, and corresponding statistics can also be found in negative sample text. The content security review system comprehensively analyzes the frequency distribution of target word tuples in both positive and negative samples to determine a discrimination threshold, namely the first threshold of 50. Finally, the content security review system compares the frequency of target word tuples in the target text with this first threshold to determine whether the target text is suspicious. It should be noted that... Figure 2 and Figure 3 The main difference lies in the granularity of the types of tuples processed. In the method of this application, the splitting of tuples is not limited to a specific character length and can include multiple types composed of at least one character.
[0056] In one optional embodiment, determining whether a target text is suspicious text based on the number of times a target word tuple appears in the target text and a first threshold includes: if the content security review system detects that the number of times a target word tuple appears in the target text is greater than or equal to the first threshold, it determines that the target text is suspicious text; if it detects that the number of times a target word tuple appears in the target text is less than the first threshold, it determines that the target text is not suspicious text and filters the target text.
[0057] Optionally, the content security review system judges text as suspicious when the number of occurrences is greater than or equal to the first threshold. The purpose of this rule is to identify text that matches the risk characteristics to a high degree, which can screen and mark potentially risky texts, providing input for subsequent more refined but computationally more resource-intensive analysis and processing steps, and is conducive to building a hierarchical processing mechanism in a resource-constrained environment.
[0058] Optionally, the content security review system determines that text appearing less than a first threshold is not suspicious and filters it. This rule aims to quickly identify and exclude text with low matching scores, filtering out a large number of low-matching texts early in the detection process. This reduces the amount of text that needs to flow into the complex downstream models, helping to lower the overall computational load and data processing latency of the content security review system. This judgment and filtering logic can serve as one of the goals of a pre-filtering mechanism: quickly removing a large amount of normal text while retaining potentially risky text as much as possible.
[0059] In one optional embodiment, determining whether a target text is suspicious based on the frequency of its occurrences within the target text includes: the content security auditing system can determine the type of the target tuple based on the number of characters it contains, wherein target tuples composed of different numbers of characters belong to different types of tuples. Then, the weight of the target tuple is determined based on its type; a weighted calculation is performed based on the weight of the target tuple and its frequency of occurrences in the target text to obtain a weighted calculation result; and the determination of whether the target text is suspicious is based on the weighted calculation result.
[0060] Optionally, the type of the target word tuple is used to represent the classification of the target word tuple based on the number of characters. For example, a target word tuple consisting of a single character belongs to one type, while a target word tuple consisting of two characters belongs to another type. The purpose of the content security review system in determining the type based on the number of characters is to distinguish feature fragments of different granularities, allowing detection methods to identify and utilize the different risk indication strengths that word tuples of different lengths may possess.
[0061] Optionally, the weights are used to represent the importance coefficients assigned to different types of target word tuples. The purpose of the weighted calculation in the content security review system is to comprehensively consider the contribution of different types of target word tuples to the final risk score. Embodiments of this application can provide a more refined evaluation mechanism, allowing target word tuples with fewer characters but potentially more universal risk indications to occupy a more significant position in the judgment. For example, a target word tuple consisting of a single character (1-gram) can be assigned a relatively high weight; a target word tuple consisting of two characters (2-gram) can be assigned a relatively low weight. The weighted calculation result can be the sum of the products of the weights of all target word tuples and their frequency of occurrence in the target text.
[0062] When determining whether a target text is suspicious based on the weighted calculation result, a judgment threshold can also be introduced. For example, a judgment threshold can be determined using prior experience. If the weighted calculation result is greater than the judgment threshold, the target text is judged to be suspicious. If the weighted calculation result is less than or equal to the judgment threshold, the target text is judged not to be suspicious.
[0063] In one optional embodiment, in determining whether the target text is suspicious text based on the weighted calculation result, another optional method is provided, including: the content security review system can calculate the ratio of the weighted calculation result to the text length of the target text to obtain a target score; if the target score is detected to be greater than a second threshold, the target text is determined to be suspicious text, wherein the second threshold is a score threshold determined using the weighted calculation method and the text length of the sample; if the target score is detected to be less than or equal to the second threshold, the target text is determined not to be suspicious text, and the target text is filtered.
[0064] Optionally, text length is used to represent the total number of characters contained in the target text. The target score is used to represent the comprehensive risk score after text length normalization. The purpose of calculating the target score in the content security audit system is to reduce or eliminate the direct impact of text length differences on the weighted calculation result. This makes the judgment criteria fairer to both long and short texts, reduces the risk of excessively long texts being misjudged simply because they appear more frequently, and reduces the risk of short texts being missed simply because they appear less frequently. This helps improve the stability of the method's judgment on texts of different lengths.
[0065] Optionally, the second threshold is another numerical benchmark used for comparison with the target score. The content security review system makes a judgment based on whether the target score is greater than the second threshold. Its purpose is to perform classification decisions on a more uniform scale that takes into account text length factors. The embodiments of this application can provide a judgment rule that may be insensitive to changes in text length. This normalization process can serve as a regularization method to eliminate the influence that factors such as text length may bring.
[0066] For example, if the target text is 100 characters long, the target score is 11 / 100 = 0.11. If the second threshold is set to 0.05, since 0.11 is greater than 0.05, the content security audit system determines that the target text is suspicious.
[0067] In one alternative embodiment, the weight of the target word tuple is negatively correlated with the number of characters included in the target word tuple.
[0068] Optionally, a negative correlation is used to represent a trend: as the number of characters in a target tuple increases, the weight assigned to that tuple by the content security auditing system tends to decrease; conversely, as the number of characters in a target tuple decreases, the weight assigned to that tuple tends to increase. The purpose of this relationship is to reflect the varying importance of features at different granularities in risk identification. During content security detection, some individual characters may possess strong independent risk orientation, while longer character combinations, although more specific, may also have an increased probability of appearing incidentally in normal text. Therefore, assigning higher weights to target tuples with fewer characters can highlight the influence of these strong risk features.
[0069] Optionally, this weighting relationship allows the detection method's judgment logic to more finely distinguish the value of different features. For example, in weighted calculations, the appearance of a high-risk single character may have a greater impact on the final target score than the appearance of a normal two-character word combination. For instance, 1-grams have a greater weight than 2-grams because 1-grams are darker; this design helps enhance the method's ability to detect texts containing only a few high-risk characters.
[0070] Optionally, the content security auditing system determines the type of the target word tuple based on the number of characters it contains. For example, a target word tuple consisting of a single character belongs to one type, called a 1-gram type, while a target word tuple consisting of two characters belongs to another type, called a 2-gram type. Then, the content security auditing system determines the weight of the target word tuple based on its type. The weight of a target word tuple is negatively correlated with the number of characters it contains; target word tuples with fewer characters (such as 1-gram types) can be assigned higher weights, while target word tuples with more characters (such as 2-gram types) can be assigned lower weights. This weighting method reflects the different risk indication strengths that different granular features can possess. This method allows for more refined judgment logic, highlighting the influence of strong risk features. The content security auditing system performs a weighted calculation based on the weight of the target word tuple and the frequency of its occurrence in the target text to obtain the weighted calculation result. For example, suppose a target text contains 3 different 1-gram target tuples and 2 different 2-gram target tuples, with a weight of 3 for 1-gram and a weight of 1 for 2-gram. The weighted calculation result would be: (1-gram count) × (weight 1) + (2-gram count) × (weight 2), which is 3 × 3 + 2 × 1 = 11. Here, the 1-gram count and 2-gram count correspond to the number of times the target tuple appears in the target text. Weight 1 and Weight 2 are weights determined by the content security review system based on the type of the target tuple. In this example, the weight (weight 1) for the 1-gram target tuple is 3, and the weight (weight 2) for the 2-gram target tuple is 1.
[0071] In an optional embodiment, the text detection method based on generalized keywords further includes: the content security review system can use either statistical method one or statistical method two to count the number of times the target word tuple appears in the target text, and to count the number of times each word tuple appears in the positive sample text and the negative sample text; wherein, statistical method one includes: when the same word tuple is detected to appear N times in the same text, the number of times the word tuple appears in the text is counted only once, where N is an integer greater than or equal to 1; wherein, statistical method two includes: when the same word tuple is detected to appear N times in the same text, the number of times the word tuple appears in the text is recorded as N times.
[0072] Optionally, statistical method one is used to characterize the deduplication counting pattern. The purpose of using statistical method one in the content security review system is to measure whether the target word tuple appears in the text, without caring about its specific frequency. Statistical method one makes the statistical results insensitive to the repeated stacking of the same risk feature in the text, and can focus more on the judgment of whether the text touches on a specific risk topic or type.
[0073] Optionally, statistical method two is used to characterize the cumulative counting pattern. The purpose of using statistical method two in the content security review system is to accumulate the total frequency of target word tuples in the text, which can reflect the degree of concentration of risk features in the text. This may be more effective in identifying texts that enhance risk intensity by repeatedly emphasizing specific words.
[0074] Optionally, the counting of word tuples in the text can include both statistical methods one and two, providing flexibility for the implementation of the solution. The content security audit system can choose statistical method one or statistical method two based on the specific needs of the application scenario or different tolerance levels for false positives and false negatives.
[0075] In at least one embodiment, the text detection method based on generalized keywords provided in this application can be applied to a variety of scenarios that require high computational efficiency, so as to give full play to its advantages as a lightweight pre-filtering mechanism.
[0076] For example, in scenarios where devices are equipped only with a central processing unit (CPU) and lack a graphics processing unit (GPU), such as edge computing devices, embedded systems, or resource-constrained servers, content security review systems struggle to directly deploy or efficiently run computationally intensive embedded models or large language models. In such environments, content security review systems can employ text detection methods based on generalized keywords as a pre-filtering approach. Since this method primarily involves keyword segmentation, frequency statistics, counting, and comparison, its computational complexity is low, making it suitable for efficient execution on a CPU. By using this method, content security review systems can quickly filter out large amounts of clearly normal text, thus improving the overall efficiency of the detection process in resource-constrained environments.
[0077] For example, in cloud deployment scenarios, even if the server has GPU resources, frequently calling the GPU to run large models for full text detection will still result in significant resource scheduling latency and economic costs. Content security moderation systems can deploy text detection methods based on generalized keywords as gateways or service front-ends. The content security moderation system first executes this method on all incoming text for rapid screening, submitting only the portions judged as suspicious to the backend GPU-based deep analysis model, such as an embedded model or a large model. This tiered processing strategy helps reduce the consumption of expensive GPU computing resources, thereby reducing system operating costs.
[0078] For example, in high-concurrency text detection scenarios, such as large social media platforms, forums, or content commenting systems that need to process massive amounts of user-generated content in real time, content security review systems can integrate text detection methods based on generalized keywords into the front end of the detection pipeline. The high processing speed and throughput of this method enable the content security review system to quickly respond to a large number of concurrent requests, initially screening out potentially risky text. This reduces the load on the more precise but slower backend detection stages, helping to improve the overall throughput and response speed of the entire content security review system.
[0079] See Figure 4 According to another aspect of the embodiments of this application, a text detection device based on generalized keywords is also provided, including: an acquisition unit 401, a first processing unit 402, a second processing unit 403 and a third processing unit 404.
[0080] The system includes: an acquisition unit 401 for acquiring a keyword set; a first processing unit 402 for splitting each keyword in the keyword set into multiple word tuples, wherein each word tuple consists of at least one character; a second processing unit 403 for filtering at least one target word tuple from all the split word tuples based on the frequency of each word tuple in the positive sample text, wherein the frequency of the target word tuple in the positive sample text is less than a preset frequency, and the preset frequencies corresponding to different types of target word tuples are different, and the preset frequency corresponding to each type of target word tuple is determined based on the statistical results of the number of times each type of target word tuple appears in the sample; and a third processing unit 404 for determining whether the target text is suspicious text based on the number of times the target word tuple appears in the target text.
[0081] Optionally, the third processing unit 404 includes: a threshold acquisition subunit for acquiring a first threshold; and a judgment subunit for judging whether the target text is suspicious text based on the number of times the target word tuple appears in the target text and the first threshold.
[0082] Optionally, the threshold acquisition subunit includes: a statistics module, used to count the number of times the target word tuple appears in each positive sample text and the number of times the target word tuple appears in each negative sample text; and a threshold determination module, used to determine a first threshold based on the number of times the target word tuple appears in each positive sample text and the number of times the target word tuple appears in each negative sample text.
[0083] Optionally, the judgment subunit includes: a first judgment module, used to judge the target text as suspicious text when the number of occurrences of the target word tuple in the target text is greater than or equal to a first threshold; and a second judgment and filtering module, used to judge the target text as not suspicious text when the number of occurrences of the target word tuple in the target text is less than the first threshold, and to filter the target text.
[0084] Optionally, the third processing unit 404 includes: a type determination subunit, used to determine the type of the target word tuple based on the number of characters included in the target word tuple, wherein target word tuples composed of different numbers of characters belong to different types of word tuples; a weight determination subunit, used to determine the weight of the target word tuple based on its type; a weighted calculation subunit, used to perform weighted calculation based on the weight of the target word tuple and the number of times the target word tuple appears in the target text, and obtain a weighted calculation result; and a result judgment subunit, used to judge whether the target text is suspicious text based on the weighted calculation result.
[0085] Optionally, the result judgment subunit includes: a score calculation module, used to calculate the ratio of the weighted calculation result to the text length of the target text to obtain the target score; a first score judgment module, used to judge the target text as suspicious text when the target score is detected to be greater than a second threshold, wherein the second threshold is a score threshold determined by using the weighted calculation method and the text length of the sample; and a second score judgment and filtering module, used to judge the target text as not suspicious text and filter the target text when the target score is detected to be less than or equal to the second threshold.
[0086] Optionally, the weight of the target word tuple determined by the weight determination subunit is negatively correlated with the number of characters included in the target word tuple.
[0087] Optionally, the text detection device based on generalized keywords further includes: a statistical configuration unit, configured to use either statistical method one or statistical method two, and to control the first processing unit 402, the second processing unit 403, and the third processing unit 404 to count the number of occurrences of target word tuples in the target text, and to count the number of occurrences of each word tuple in the positive sample text and the negative sample text; wherein, statistical method one includes: when the same word tuple is detected to appear N times in the same text, the number of occurrences of the word tuple in the text is counted only once, where N is an integer greater than or equal to 1; wherein, statistical method two includes: when the same word tuple is detected to appear N times in the same text, the number of occurrences of the word tuple in the text is recorded as N times.
[0088] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the above-described text detection method based on generalized keywords.
[0089] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described text detection method based on generalized keywords.
[0090] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described text detection method based on generalized keywords.
[0091] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0092] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0093] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0097] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A text detection method based on generalized keywords, characterized in that, include: Get the keyword set; Each keyword in the keyword set is split into multiple word tuples, wherein each word tuple consists of at least one character; Based on the frequency of each word tuple in the positive sample text, at least one target word tuple is selected from all the split word tuples. The frequency of the target word tuple in the positive sample text is less than a preset frequency. Different types of target word tuples correspond to different preset frequencies, and the preset frequency corresponding to each type of target word tuple is determined based on the statistical results of the number of times each type of target word tuple appears in the sample. Based on the number of times the target word tuple appears in the target text, it is determined whether the target text is suspicious.
2. The text detection method based on generalized keywords according to claim 1, characterized in that, Based on the frequency of occurrence of the target word tuple in the target text, determine whether the target text is suspicious, including: Obtain the first threshold; Based on the number of times the target word tuple appears in the target text and the first threshold, it is determined whether the target text is the suspicious text.
3. The text detection method based on generalized keywords according to claim 2, characterized in that, Obtaining the first threshold includes: The number of times the target word tuple appears in each positive sample text and the number of times the target word tuple appears in each negative sample text are counted. The first threshold is determined based on the number of times the target word tuple appears in each positive sample text and the number of times the target word tuple appears in each negative sample text.
4. The text detection method based on generalized keywords according to claim 2, characterized in that, Determining whether the target text is the suspicious text based on the frequency of occurrence of the target word tuple in the target text and the first threshold includes: If the number of occurrences of the target word tuple in the target text is greater than or equal to the first threshold, the target text is determined to be suspicious text. If the number of occurrences of the target word tuple in the target text is less than the first threshold, the target text is determined not to be suspicious text, and the target text is filtered.
5. The text detection method based on generalized keywords according to claim 1, characterized in that, Based on the frequency of occurrence of the target word tuple in the target text, determine whether the target text is suspicious, including: The type of the target word tuple is determined based on the number of characters it contains, wherein target word tuples composed of different numbers of characters belong to different types of word tuples; The weight of the target word tuple is determined based on its type. A weighted calculation is performed based on the weight of the target word tuple and the number of times the target word tuple appears in the target text to obtain the weighted calculation result; The weighted calculation result is used to determine whether the target text is suspicious.
6. The text detection method based on generalized keywords according to claim 5, characterized in that, Determining whether the target text is suspicious based on the weighted calculation result includes: The target score is obtained by calculating the ratio of the weighted calculation result to the text length of the target text. If the target score is found to be greater than the second threshold, the target text is determined to be suspicious text, wherein the second threshold is a score threshold determined by using a weighted calculation method and the text length of the sample; If the target score is detected to be less than or equal to the second threshold, the target text is determined not to be suspicious text, and the target text is filtered.
7. The text detection method based on generalized keywords according to claim 5, characterized in that, The weight of the target word tuple is negatively correlated with the number of characters included in the target word tuple.
8. The text detection method based on generalized keywords according to any one of claims 1 to 7, characterized in that, The text detection method based on generalized keywords also includes: Using either statistical method one or statistical method two, the number of times the target word tuple appears in the target text is counted, as well as the number of times each word tuple appears in the positive sample text and the negative sample text. The first statistical method includes: when the same word tuple appears N times in the same text, the number of times the word tuple appears in the text is counted only once, where N is an integer greater than or equal to 1; The second statistical method includes: when the same word tuple appears N times in the same text, the number of times the word tuple appears in the text is recorded as N times.
9. A text detection device based on generalized keywords, characterized in that, include: The retrieval unit is used to retrieve a set of keywords; The first processing unit is configured to split each keyword in the keyword set into multiple word tuples, wherein each word tuple consists of at least one character; The second processing unit is used to filter at least one target word group from all the split word groups according to the frequency of each word group in the positive sample text. The frequency of the target word group in the positive sample text is less than a preset frequency. The preset frequencies corresponding to different types of target word groups are different, and the preset frequency corresponding to each type of target word group is determined based on the statistical results of the number of times each type of target word group appears in the sample. The third processing unit is used to determine whether the target text is suspicious text based on the number of times the target word tuple appears in the target text.
10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the text detection method based on generalized keywords as described in any one of claims 1 to 8.