Text detection methods, devices, electronic equipment and storage media

Abstract

This invention provides a text detection method, which includes: acquiring a text to be detected; performing sensitive word detection processing on the text to be detected based on a preset sensitive word tree to obtain a first sensitive word set and a first non-sensitive word set; performing sensitive word detection processing on the text to be detected based on a preset large language model to obtain a second sensitive word set and a second non-sensitive word set; and determining whether the text to be detected contains sensitive words based on the first sensitive word set, the first non-sensitive word set, the second sensitive word set, and the second non-sensitive word set. Due to the characteristics of the large language model, the sensitive word detection processing can take into account the inherent semantic features of the text. Therefore, by integrating the second sensitive word set and the second non-sensitive word set related to the inherent semantic features, as well as the first sensitive word set and the first non-sensitive word set related to the sensitive word tree, the accuracy of text detection can be improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and more particularly to a text detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the increasing prevalence of text-based information technology and text-based natural language processing, the importance of text security detection in engineering applications is becoming increasingly prominent. However, existing text security detection methods mainly rely on hard matching techniques with fixed lexicons. This method often ignores the inherent semantic features of text, resulting in low accuracy in text security detection and thus increasing security risks. Therefore, how to provide a text detection method that can improve the accuracy of text security detection has become an urgent problem to be solved. Summary of the Invention

[0003] This invention provides a text detection method aimed at addressing the low accuracy of existing text detection methods in text security detection. By inputting the text to be detected into a sensitive word tree and a large language model for sensitive word detection processing, corresponding first sensitive word set, first non-sensitive word set, second sensitive word set, and second non-sensitive word set are obtained. Due to the characteristics of the large language model, the inherent semantic features of the text can be considered during sensitive word detection processing. Therefore, by integrating the second sensitive word set and second non-sensitive word set related to inherent semantic features, as well as the first sensitive word set and first non-sensitive word set related to the sensitive word tree, it is possible to more accurately determine whether the text to be detected contains sensitive words.

[0004] In a first aspect, embodiments of the present invention provide a text detection method, the method comprising the following steps:

[0005] Obtain the text to be detected;

[0006] Based on a preset sensitive word tree, the text to be detected is processed to detect sensitive words, resulting in a first set of sensitive words and a first set of non-sensitive words;

[0007] Based on a preset large language model, the text to be detected is processed to detect sensitive words, resulting in a second set of sensitive words and a second set of non-sensitive words.

[0008] Based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words, it is determined whether the text to be detected contains sensitive words.

[0009] Optionally, determining whether the text to be detected contains sensitive words based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words includes:

[0010] Calculate the first average confidence level of all sensitive words in the first set of sensitive words;

[0011] Calculate the second average confidence level of all non-sensitive words in the first set of non-sensitive words;

[0012] Calculate the third average confidence level of all sensitive words in the second set of sensitive words;

[0013] Calculate the fourth average confidence level of all non-sensitive words in the second set of non-sensitive words;

[0014] Based on the first average confidence level, the second average confidence level, the third average confidence level, and the fourth average confidence level, it is determined whether the text to be detected contains the sensitive word.

[0015] Optionally, determining whether the text to be detected contains the sensitive word based on the first average confidence level, the second average confidence level, the third average confidence level, and the fourth average confidence level includes:

[0016] A first fused confidence level is obtained by fusing the first average confidence level and the third average confidence level.

[0017] A second fused confidence level is obtained by fusing the second average confidence level and the fourth average confidence level.

[0018] Based on the first fusion confidence score and the second fusion confidence score, it is determined whether the text to be detected contains the sensitive word.

[0019] Optionally, determining whether the sensitive word is contained in the text to be detected based on the first fusion confidence score and the second fusion confidence score includes:

[0020] Based on the first fusion confidence score and the second fusion confidence score, the target label confidence score of the text to be detected is determined;

[0021] Based on the confidence level of the target label, it is determined whether the text to be detected contains the sensitive word.

[0022] Optionally, determining the target label confidence of the text to be detected based on the first fusion confidence and the second fusion confidence includes:

[0023] When the first fusion confidence score is greater than the second fusion confidence score, the first fusion confidence score is determined as the target label confidence score;

[0024] When the first fusion confidence score is less than the second fusion confidence score, the second fusion confidence score is determined as the target label confidence score;

[0025] When the first fusion confidence level is equal to the second fusion confidence level, the target label confidence level is empty.

[0026] Optionally, determining whether the text to be detected contains the sensitive word based on the target label confidence score includes:

[0027] When the confidence level of the target label is greater than the preset confidence level threshold, it is determined whether the text to be detected contains the sensitive word based on the confidence level of the target label.

[0028] When the confidence level of the target label is not greater than the preset confidence level threshold or the confidence level of the target label is empty, the text to be detected is fed back to the relevant personnel corresponding to the text to be detected, so that the relevant personnel can review the text to be detected and determine whether the text to be detected contains the sensitive words.

[0029] Optionally, after the step of sending the text to be detected to the relevant personnel so that they can review the text and determine whether it contains the sensitive word, the method further includes:

[0030] When the text to be detected contains the sensitive word, the target sensitive word is extracted from the text to be detected;

[0031] Based on the target sensitive words, the preset sensitive word tree is updated.

[0032] Secondly, embodiments of the present invention also provide a text detection device, the text detection device comprising:

[0033] The first acquisition module is used to acquire the text to be detected;

[0034] The first detection module is used to perform sensitive word detection processing on the text to be detected based on a preset sensitive word tree, so as to obtain a first set of sensitive words and a first set of non-sensitive words;

[0035] The second detection module is used to perform sensitive word detection processing on the text to be detected based on a preset large language model, so as to obtain a second set of sensitive words and a second set of non-sensitive words.

[0036] The first determining module is used to determine whether the text to be detected contains sensitive words based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words.

[0037] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the text detection method provided in embodiments of the present invention.

[0038] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the text detection method provided in the embodiments of the present invention.

[0039] In this embodiment of the invention, a text to be detected is obtained; the text to be detected is processed for sensitive word detection based on a preset sensitive word tree to obtain a first set of sensitive words and a first set of non-sensitive words; the text to be detected is processed for sensitive word detection based on a preset large language model to obtain a second set of sensitive words and a second set of non-sensitive words; based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words, it is determined whether the text to be detected contains sensitive words. By inputting the text to be detected into the sensitive word tree and the large language model respectively for sensitive word detection processing, the corresponding first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words are obtained. Due to the characteristics of the large language model, the sensitive word detection processing can take into account the inherent semantic features of the text. Therefore, after integrating the second set of sensitive words and the second set of non-sensitive words related to the inherent semantic features, as well as the first set of sensitive words and the first set of non-sensitive words related to the sensitive word tree, it is possible to more accurately determine whether the text to be detected contains sensitive words. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of a text detection method provided in an embodiment of the present invention;

[0042] Figure 2 This is a flowchart of another text detection method provided in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of a text detection device provided in an embodiment of the present invention;

[0044] Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, Figure 1 This is a flowchart of a text detection method provided in an embodiment of the present invention, including:

[0047] 101. Obtain the text to be detected.

[0048] In this embodiment of the invention, the above-described text detection method can be applied to a text detection platform. This platform can be constructed using a server or server cluster. The server or server cluster can be any electronic device with functions such as data analysis, data processing, data transmission, and data storage. The text to be detected can be uploaded to the text detection platform via the data transmission function. Upon receiving the text to be detected, the platform can perform text detection using any of the text detection methods provided in this embodiment to determine whether the text contains sensitive words. The text to be detected can consist of at least one text unit, which can be a character, word, symbol, etc.

[0049] 102. Based on the preset sensitive word tree, perform sensitive word detection processing on the text to be detected to obtain the first set of sensitive words and the first set of non-sensitive words.

[0050] In this embodiment of the invention, the aforementioned sensitive word tree can be constructed using a prefix tree, which can be understood as a trie, a tree-like data structure used to store a set of strings. Specifically, the sensitive word library can be constructed using the aforementioned prefix tree to obtain the sensitive word tree. For example, the nodes of the aforementioned prefix tree can be initialized, and each character in the sensitive word library can be inserted into the prefix tree as a node. During the insertion process, if the character already exists on the path, the path is followed downwards; otherwise, a new node needs to be created. If the inserted character is the last character of a sensitive word, this node can be marked as the end node to indicate that the node represents a complete sensitive word. This process continues until all sensitive words are inserted into the aforementioned prefix tree, resulting in the aforementioned sensitive word tree, where each path represents a sensitive word.

[0051] Specifically, after constructing the aforementioned sensitive word tree, characters from the text to be detected are inserted one by one into the prefix tree, and it is checked whether they match the sensitive words in the sensitive word tree. That is, if the current character has a corresponding node in the tree, the matching continues to the next character; if the path continues to match to the end node of a sensitive word, it means that a sensitive word has been detected and recorded. If no matching node is found, the current check is skipped, and the matching continues from the next character.

[0052] After matching is completed (i.e., after sensitive word detection processing is completed), a first set of sensitive words and a first set of non-sensitive words are obtained. The first set of sensitive words includes at least one first sensitive word and its corresponding confidence level. The first set of non-sensitive words includes at least one first non-sensitive word and its corresponding confidence level. The confidence level can be determined by at least one criterion, such as word frequency, path depth, contextual information, and historical false alarm rate. Both the first sensitive words and the first non-sensitive words can exist in the form of labeled data, used to mark whether a word in the text to be detected is a sensitive word or a non-sensitive word.

[0053] After the above sensitive word tree matching is completed, the data structure of its output result can be [A] = [α1, α2, α3..., α...]. n-1 ,α n ], where A represents the above output result. In the above output result, a1, a2, etc. can be the confidence level of a sensitive word or a non-sensitive word. Add the corresponding sensitive words to the first sensitive word set and the non-sensitive words to the first non-sensitive word set. After all sensitive words and non-sensitive words are added, the above first sensitive word set and first non-sensitive word set can be obtained.

[0054] 103. Based on the preset large language model, perform sensitive word detection processing on the text to be detected to obtain a second set of sensitive words and a second set of non-sensitive words.

[0055] In this embodiment of the invention, the aforementioned preset Large Language Model (LLM) can be any deep learning-based natural language processing model, such as chatgpt, Wenxin Yiyan Large Model, Xinghuo Large Model, etc. These large language models possess a large number of parameters and powerful language understanding and generation capabilities. They are obtained by training on a large amount of text data and are capable of completing the required language tasks (i.e., including sensitive word detection).

[0056] Specifically, the text to be detected can be added to a preset prompt template to obtain prompt text. This prompt text is then provided to the preset large language model, which performs sensitive word detection on the text to be detected, resulting in a second set of sensitive words and a second set of non-sensitive words. The second set of sensitive words includes at least one second sensitive word and its corresponding confidence level. The second set of non-sensitive words includes at least one second non-sensitive word and its corresponding confidence level. Both the second sensitive words and the second non-sensitive words can exist as labeled data, used to mark whether a word in the text to be detected is a sensitive word or a non-sensitive word.

[0057] After completing the above sensitive word detection process, the pre-defined large language model will output a result, the data structure of which can be:

[0058]

[0059] Wherein, B represents the above output result, where This is represented as the single-label confidence score of the first second sensitive word or the second non-sensitive word. is the single-label confidence penalty coefficient for the first second sensitive word or the second non-sensitive word, where i represents the number of categories in the label. Add the corresponding sensitive words to the second sensitive word set and the non-sensitive words to the second non-sensitive word set. After all sensitive and non-sensitive words have been added, the aforementioned second sensitive word set and second non-sensitive word set are obtained.

[0060] 104. Based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words, determine whether the text to be detected contains sensitive words.

[0061] In this embodiment of the invention, the average confidence level of sensitive words or non-sensitive words in each set can be calculated, and the average confidence level of the first set of sensitive words and the average confidence level of the second set of sensitive words can be fused (for example, a weighted calculation method can be used) to obtain the first fused confidence level. The average confidence levels of the first set of non-sensitive words and the second set of non-sensitive words can be fused (for example, a weighted calculation method can be used) to obtain the second fused confidence level.

[0062] The first fusion confidence score is compared with the second fusion confidence score. If the first fusion confidence score is greater than the second fusion confidence score, the text to be detected contains sensitive words. If the first fusion confidence score is less than the second fusion confidence score, the text to be detected does not contain sensitive words.

[0063] In this embodiment of the invention, a text to be detected is obtained; the text to be detected is processed for sensitive word detection based on a preset sensitive word tree to obtain a first set of sensitive words and a first set of non-sensitive words; the text to be detected is processed for sensitive word detection based on a preset large language model to obtain a second set of sensitive words and a second set of non-sensitive words; based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words, it is determined whether the text to be detected contains sensitive words. By inputting the text to be detected into the sensitive word tree and the large language model respectively for sensitive word detection processing, the corresponding first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words are obtained. Due to the characteristics of the large language model, the sensitive word detection processing can take into account the inherent semantic features of the text. Therefore, after integrating the second set of sensitive words and the second set of non-sensitive words related to the inherent semantic features, as well as the first set of sensitive words and the first set of non-sensitive words related to the sensitive word tree, it is possible to more accurately determine whether the text to be detected contains sensitive words.

[0064] It is understood that in the specific implementation of this application, data such as the text to be detected are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data, as well as the construction, training and use of related models, must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0065] It should be noted that the text detection method provided in this embodiment of the invention can be applied to computers, servers and other devices capable of text detection.

[0066] Optionally, in the step of determining whether the text to be detected contains sensitive words based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words, the following steps may be taken: calculating the first average confidence score of all sensitive words in the first set of sensitive words; calculating the second average confidence score of all non-sensitive words in the first set of non-sensitive words; calculating the third average confidence score of all sensitive words in the second set of sensitive words; calculating the fourth average confidence score of all non-sensitive words in the second set of non-sensitive words; and determining whether the text to be detected contains sensitive words based on the first average confidence score, the second average confidence score, the third average confidence score, and the fourth average confidence score.

[0067] In this embodiment of the invention, the first sensitive word set includes a first sensitive word (i.e., a tag, a first sensitive word tag), which corresponds to a tag confidence level; the first non-sensitive word set includes a first non-sensitive word (i.e., a tag, a first non-sensitive word tag), which also corresponds to a tag confidence level; the second sensitive word set includes a second sensitive word (i.e., a tag, a second sensitive word tag), which corresponds to a tag confidence level; and the second non-sensitive word set includes a second non-sensitive word (i.e., a tag, a second non-sensitive word tag), which also corresponds to a tag confidence level.

[0068] Specifically, the sum of the confidence scores of all labels in each set can be counted to calculate the average confidence score for each set.

[0069] More specifically, the first and second average confidence levels mentioned above can be calculated using the following formulas:

[0070]

[0071] Among them, the above Represented as the first average confidence level or the second average confidence level, the above α i denoted as the confidence score of a subset (i.e., the first set of sensitive words or the second set of sensitive words), where i represents the number of label categories (i.e., the number of first sensitive words in the first set of sensitive words or the number of second sensitive words in the second set of sensitive words).

[0072] The third and fourth average confidence levels mentioned above can be calculated using the following formulas:

[0073]

[0074] Wherein, the above λ i Represented as the third or fourth average confidence level, the above ε i This represents the single-label confidence level (i.e., the label confidence level of the second sensitive word in the second sensitive word set or the label confidence level of the second non-sensitive word in the second non-sensitive word set), as mentioned above. i denoted as the single-label confidence penalty coefficient, where i represents the number of categories in the label (i.e., the number of second sensitive words in the second sensitive word set or the number of second non-sensitive words in the second non-sensitive word set).

[0075] After calculating the first, second, third, and fourth average confidence levels, the first and third average confidence levels can be fused together to obtain the first fused confidence level, and the second and fourth average confidence levels can be fused together to obtain the second fused confidence level. The first and second fused confidence levels are then compared to obtain the comparison results. Based on the comparison results, it is determined whether the text to be detected contains sensitive words.

[0076] Alternatively, the first average confidence level can be directly compared with the second average confidence level to obtain the first comparison result, and the third average confidence level can be compared with the fourth average confidence level to obtain the second comparison result.

[0077] If the first comparison result shows that the first average confidence level is greater than the second average confidence level, and the third average confidence level is also greater than the fourth average confidence level, then the text to be detected contains sensitive words. Conversely, if the first comparison result shows that the first average confidence level is less than the second average confidence level, and the third average confidence level is also less than the fourth average confidence level, then the text to be detected does not contain sensitive words. If the first comparison result differs from the second comparison result—for example, if the first comparison result shows that the first average confidence level is greater than the second average confidence level, and the second comparison result shows that the third average confidence level is less than the fourth average confidence level—then the text to be detected can be sent to the administrators of the text detection platform for review to determine whether it contains sensitive words.

[0078] Optionally, in the step of determining whether the text to be detected contains sensitive words based on the first average confidence level, the second average confidence level, the third average confidence level, and the fourth average confidence level, a fusion process can be performed based on the first average confidence level and the third average confidence level to obtain a first fused confidence level; a fusion process can be performed based on the second average confidence level and the fourth average confidence level to obtain a second fused confidence level; and a determination can be made based on the first fused confidence level and the second fused confidence level to determine whether the text to be detected contains sensitive words.

[0079] In this embodiment of the invention, a first confidence weight can be set for the output confidence of the sensitive word trees corresponding to the first average confidence and the second average confidence. Simultaneously, a second confidence weight can be set for the output confidence of the large language model corresponding to the third average confidence and the fourth average confidence.

[0080] During the fusion process, the first average confidence level can be calculated by weighting the first confidence level according to the first confidence level weight, and the third average confidence level can be calculated by weighting the third confidence level according to the second confidence level weight. The first average confidence level after weighting the first average confidence level and the third average confidence level are added together to obtain the first fusion confidence level.

[0081] Similarly, the second average confidence level is calculated by weighting the first confidence level, and the fourth average confidence level is calculated by weighting the second confidence level. The weighted second average confidence level and the fourth average confidence level are then added together to obtain the second fusion confidence level.

[0082] After calculating the first fusion confidence and the second fusion confidence, the first fusion confidence and the second fusion confidence can be compared to obtain the corresponding comparison results. Based on the comparison results, it can be determined whether the text to be detected contains sensitive words.

[0083] Specifically, the first fusion confidence score and the second fusion confidence score mentioned above can be calculated using the following formula:

[0084]

[0085] Among them, the above r i Represented as the first fusion confidence level or the second fusion confidence level, the above w m The first confidence weight mentioned above is represented by w. n This is represented as the second confidence weight, as mentioned above. This is represented by the first average confidence level or the second average confidence level mentioned above, where λ is... i This is represented as the third or fourth average confidence level. It should be noted that since the second confidence weight corresponds to the aforementioned large language model, and the large language model is more semantically biased than the sensitive word tree when detecting sensitive words, to improve the accuracy of text detection, the second confidence weight can be set to be greater than the first confidence weight. For example, the first confidence weight can be set to 0.4, and the second confidence weight to 0.6. The specific requirement that the second confidence weight should be greater than the first confidence weight (i.e., the weight ratio) can be determined based on historical text detection experience or through a limited number of trials.

[0086] Optionally, in the step of determining whether the text to be detected contains sensitive words based on the first fusion confidence and the second fusion confidence, the target label confidence of the text to be detected can also be determined based on the first fusion confidence and the second fusion confidence; and the text to be detected can be determined based on the target label confidence.

[0087] In this embodiment of the invention, when the first fusion confidence level is greater than the second fusion confidence level, the first fusion confidence level can be determined as the target label confidence level; when the first fusion confidence level is less than the second fusion confidence level, the second fusion confidence level can be determined as the target label confidence level. Furthermore, based on the aforementioned target label confidence level, it can be quickly determined whether the text to be detected contains sensitive words. That is, when the target label confidence level is the first fusion confidence level, it can be quickly determined that the text to be detected contains sensitive words; when the target label confidence level is the second fusion confidence level, it can be quickly determined that the text to be detected does not contain sensitive words.

[0088] Optionally, in the step of determining the target label confidence of the text to be detected based on the first fusion confidence and the second fusion confidence, the first fusion confidence can be determined as the target label confidence when the first fusion confidence is greater than the second fusion confidence; the second fusion confidence can be determined as the target label confidence when the first fusion confidence is less than the second fusion confidence; and the target label confidence is empty when the first fusion confidence is equal to the second fusion confidence.

[0089] In this embodiment of the invention, when the first fusion confidence score is greater than the second fusion confidence score, it can be simply understood that both the aforementioned sensitive word tree and the aforementioned language model believe that the confidence score of the text to be detected containing sensitive words is greater than the confidence score of the text not containing sensitive words. Therefore, the first fusion confidence score can be determined as the target label confidence score. Conversely, when the first fusion confidence score is less than the second fusion confidence score, it can be simply understood that both the aforementioned sensitive word tree and the aforementioned language model believe that the confidence score of the text to be detected containing sensitive words is less than the confidence score of the text not containing sensitive words. Therefore, the second fusion confidence score can be determined as the target label confidence score.

[0090] If a special case occurs where the first fusion confidence score equals the second fusion confidence score, it can be simply understood that both the sensitive word tree and the large language model believe that the confidence score of the text to be detected containing sensitive words is equal to the confidence score of the text not containing sensitive words. In this case, it is impossible to determine whether the text to be detected contains sensitive words, so the confidence score of the target label can be set to empty.

[0091] Optionally, in the step of determining whether the text to be detected contains sensitive words based on the confidence level of the target label, if the confidence level of the target label is greater than a preset confidence threshold, the text to be detected can be determined based on the confidence level of the target label; if the confidence level of the target label is not greater than the preset confidence threshold or the confidence level of the target label is empty, the text to be detected can be fed back to the relevant personnel corresponding to the text to be detected, so that the relevant personnel can review the text to be detected and determine whether the text to be detected contains sensitive words.

[0092] In this embodiment of the invention, the preset confidence threshold d can be obtained through historical text detection experience or through a limited number of experiments. For example, it can be set to 0.6. When the confidence of the target label is greater than the preset confidence threshold, it indicates that the label corresponding to the confidence of the target label is credible. At this time, if the corresponding label is a sensitive word, the text to be detected contains a sensitive word; if the corresponding label is a non-sensitive word, the text to be detected does not contain a sensitive word.

[0093] Correspondingly, if the confidence level of the target label is not greater than (i.e. less than or equal to) the preset confidence level threshold, it means that the label corresponding to the confidence level of the target label is untrustworthy. If the confidence level of the target label is empty, it means that the first fusion confidence level is equal to the second fusion confidence level. In both of these cases, it is impossible to determine whether the text to be detected contains sensitive words. Therefore, at this time, the text to be detected can be fed back to the relevant personnel, such as the administrators of the aforementioned text detection platform, so that the relevant personnel can review the text to be detected and determine whether the text to be detected contains sensitive words.

[0094] Optionally, after the step of sending the text to be detected to the relevant personnel so that they can review it and determine whether it contains sensitive words, if the text contains sensitive words, the target sensitive words can be extracted from the text; and the preset sensitive word tree can be updated based on the target sensitive words.

[0095] In this embodiment of the invention, since it is impossible to determine whether the text to be detected contains sensitive words in either the case where the confidence level of the target label is not greater than a preset confidence threshold, or the case where the confidence level of the target label is empty, the target sensitive words can be extracted from the text after relevant personnel have reviewed the text to be detected and determined whether it contains sensitive words. This extraction can be performed by the aforementioned relevant personnel, or by other sensitive word trees or other large language models. After extraction, the preset sensitive word tree can be updated based on the target sensitive words to improve the generalization ability of the sensitive word tree and further improve the accuracy of subsequent text detection.

[0096] like Figure 2 As shown in the diagram, this embodiment of the invention also provides a flowchart of another text detection method. This includes the following steps:

[0097] Step 1 is the process of building the sensitive word tree. Specifically, it may involve obtaining a set of sensitive words, building a sensitive word tree based on the set of sensitive words, and loading the sensitive word tree into the platform's memory for quick access.

[0098] Step 2 is the sensitive word detection process. Specifically, the input text to be detected can be input into the sensitive word tree and the large language model (i.e., the language model semantic detection model) respectively for sensitive word detection processing to obtain the corresponding first sensitive word set, first non-sensitive word set, second sensitive word set and second non-sensitive word set;

[0099] Step 3 involves the fusion process of the confidence scores of sensitive word tags in the sensitive word set. Specifically, this can be achieved by calculating the average confidence score of each set, namely, the first average confidence score, the second average confidence score, the third average confidence score, and the fourth average confidence score. The first average confidence score is then fused with the third average confidence score to obtain the first fused confidence score, and the second average confidence score is then fused with the fourth average confidence score to obtain the second fused confidence score. Finally, the confidence score of the target tag is determined from the first fused confidence score and the second fused confidence score.

[0100] Step 4 is the threshold judgment process. Specifically, the confidence level of the target label is compared with the preset confidence threshold. If the confidence level of the target label is greater than the preset confidence threshold, the label (i.e., sensitive word or non-sensitive word) corresponding to the confidence level of the target label can be output. Otherwise, the text to be detected can be fed back to the relevant personnel mentioned above.

[0101] Step 5 is the manual review process. When relevant personnel receive the text to be tested, they can review it to determine whether it is safe text (i.e., whether it contains sensitive words). If it is safe text, the result can be output. If it is not safe text, the target sensitive words can be extracted from the text to be tested.

[0102] Step 6 is the process of updating the sensitive word tree. After identifying the target sensitive words, they are fed back to Step 1, i.e., added to the sensitive word set, resulting in a new sensitive word set. A new sensitive word tree is then constructed based on this new set and added to memory. This new sensitive word tree is used for subsequent text detection processing of other texts to be detected, further improving the accuracy of text detection.

[0103] Understandably, this invention utilizes a dynamic construction of a sensitive word tree combined with an advanced large language model for qualitative and quantitative analysis of the text to be detected. Compared to traditional text matching methods based on fixed vocabularies, this invention significantly improves the accuracy of text security detection, effectively reducing the possibility of false positives and false negatives, and providing an innovative solution for the field of text security detection. It also provides a basis for subsequent security measures. Furthermore, this invention not only improves the accuracy of text security detection but also possesses high adaptability and flexibility, capable of addressing various complex text security challenges. Through this method, users can be effectively protected from malicious text content, providing crucial technical support for the development of the text security detection field.

[0104] like Figure 3 As shown, embodiments of the present invention also provide a text detection device, comprising:

[0105] The first acquisition module 301 is used to acquire the text to be detected;

[0106] The first detection module 302 is used to perform sensitive word detection processing on the text to be detected based on a preset sensitive word tree, so as to obtain a first set of sensitive words and a first set of non-sensitive words;

[0107] The second detection module 303 is used to perform sensitive word detection processing on the text to be detected based on a preset large language model, so as to obtain a second set of sensitive words and a second set of non-sensitive words.

[0108] The first determining module 304 is used to determine whether the text to be detected contains sensitive words based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words.

[0109] Optionally, the first determining module 304 includes:

[0110] The first calculation submodule is used to calculate the first average confidence level of all sensitive words in the first sensitive word set;

[0111] The second calculation submodule is used to calculate the second average confidence level of all non-sensitive words in the first non-sensitive word set;

[0112] The third calculation submodule is used to calculate the third average confidence level of all sensitive words in the second set of sensitive words;

[0113] The fourth calculation submodule is used to calculate the fourth average confidence level of all non-sensitive words in the second set of non-sensitive words;

[0114] The first determining submodule is used to determine whether the text to be detected contains the sensitive word based on the first average confidence level, the second average confidence level, the third average confidence level, and the fourth average confidence level.

[0115] Optionally, the first determining submodule includes:

[0116] The first fusion unit is used to perform fusion processing based on the first average confidence level and the third average confidence level to obtain a first fusion confidence level;

[0117] The second fusion unit is used to perform fusion processing based on the second average confidence level and the fourth average confidence level to obtain a second fusion confidence level;

[0118] The first determining unit is configured to determine whether the text to be detected contains the sensitive word based on the first fusion confidence and the second fusion confidence.

[0119] Optionally, the first determining unit includes:

[0120] The first determining subunit is used to determine the target label confidence of the text to be detected based on the first fusion confidence and the second fusion confidence;

[0121] The second determining subunit is used to determine whether the text to be detected contains the sensitive word based on the confidence level of the target label.

[0122] Optionally, the first determining subunit is further configured to determine the first fusion confidence as the target label confidence when the first fusion confidence is greater than the second fusion confidence;

[0123] The first determining subunit is further configured to determine the second fusion confidence as the target label confidence when the first fusion confidence is less than the second fusion confidence;

[0124] The first determining subunit is further configured to set the target label confidence level to empty when the first fusion confidence level is equal to the second fusion confidence level.

[0125] Optionally, the second determining subunit is further configured to determine whether the text to be detected contains the sensitive word based on the target label confidence level when the confidence level of the target label is greater than a preset confidence level threshold;

[0126] The second determining subunit is further configured to, when the confidence level of the target label is not greater than a preset confidence level threshold or the confidence level of the target label is empty, feed back the text to be detected to the relevant personnel corresponding to the text to be detected, so that the relevant personnel can review the text to be detected and determine whether the text to be detected contains the sensitive words.

[0127] Optionally, the text detection device further includes:

[0128] The first extraction subunit is used to extract the target sensitive word from the text to be detected when the text to be detected contains the sensitive word.

[0129] The first update subunit is used to update the preset sensitive word tree based on the target sensitive word.

[0130] like Figure 4 As shown, this embodiment of the invention also provides an electronic device, characterized in that it includes a processor, which can execute any of the above-described text detection methods.

[0131] Specifically, it includes a processor 401 and a memory 402, as well as a computer program stored in the memory 402 and capable of running on the processor 401 to perform a text detection method, wherein:

[0132] The processor 401 executes the calculator program containing the text detection method stored in the memory 402, and performs the following steps:

[0133] Obtain the text to be detected;

[0134] Based on a preset sensitive word tree, the text to be detected is processed to detect sensitive words, resulting in a first set of sensitive words and a first set of non-sensitive words;

[0135] Based on a preset large language model, the text to be detected is processed to detect sensitive words, resulting in a second set of sensitive words and a second set of non-sensitive words.

[0136] Based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words, it is determined whether the text to be detected contains sensitive words.

[0137] Optionally, the step of determining whether the text to be detected contains sensitive words based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words, executed by the processor 401, includes:

[0138] Calculate the first average confidence level of all sensitive words in the first set of sensitive words;

[0139] Calculate the second average confidence level of all non-sensitive words in the first set of non-sensitive words;

[0140] Calculate the third average confidence level of all sensitive words in the second set of sensitive words;

[0141] Calculate the fourth average confidence level of all non-sensitive words in the second set of non-sensitive words;

[0142] Based on the first average confidence level, the second average confidence level, the third average confidence level, and the fourth average confidence level, it is determined whether the text to be detected contains the sensitive word.

[0143] Optionally, the process executed by processor 401 to determine whether the text to be detected contains the sensitive word based on the first average confidence level, the second average confidence level, the third average confidence level, and the fourth average confidence level includes:

[0144] A first fused confidence level is obtained by fusing the first average confidence level and the third average confidence level.

[0145] A second fused confidence level is obtained by fusing the second average confidence level and the fourth average confidence level.

[0146] Based on the first fusion confidence score and the second fusion confidence score, it is determined whether the text to be detected contains the sensitive word.

[0147] Optionally, the process executed by processor 401 to determine whether the text to be detected contains the sensitive word based on the first fusion confidence score and the second fusion confidence score includes:

[0148] Based on the first fusion confidence score and the second fusion confidence score, the target label confidence score of the text to be detected is determined;

[0149] Based on the confidence level of the target label, it is determined whether the text to be detected contains the sensitive word.

[0150] Optionally, the process of determining the target label confidence of the text to be detected based on the first fusion confidence and the second fusion confidence, performed by the processor 401, includes:

[0151] When the first fusion confidence score is greater than the second fusion confidence score, the first fusion confidence score is determined as the target label confidence score;

[0152] When the first fusion confidence score is less than the second fusion confidence score, the second fusion confidence score is determined as the target label confidence score;

[0153] When the first fusion confidence level is equal to the second fusion confidence level, the target label confidence level is empty.

[0154] Optionally, the process performed by processor 401 to determine whether the text to be detected contains the sensitive word based on the target label confidence score includes:

[0155] When the confidence level of the target label is greater than the preset confidence level threshold, it is determined whether the text to be detected contains the sensitive word based on the confidence level of the target label.

[0156] When the confidence level of the target label is not greater than the preset confidence level threshold or the confidence level of the target label is empty, the text to be detected is fed back to the relevant personnel corresponding to the text to be detected, so that the relevant personnel can review the text to be detected and determine whether the text to be detected contains the sensitive words.

[0157] Optionally, after the process of sending the text to be detected to the relevant personnel so that they can review the text and determine whether it contains the sensitive word, the method executed by the processor 401 further includes:

[0158] When the text to be detected contains the sensitive word, the target sensitive word is extracted from the text to be detected;

[0159] Based on the target sensitive words, the preset sensitive word tree is updated.

[0160] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the text detection method or the application-side text detection method provided in this invention, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0161] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The computer-readable storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0162] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A text detection method, characterized in that, The method includes the following steps: Obtain the text to be detected; Based on a preset sensitive word tree, the text to be detected is processed to detect sensitive words, resulting in a first set of sensitive words and a first set of non-sensitive words; Based on a preset large language model, the text to be detected is processed to detect sensitive words, resulting in a second set of sensitive words and a second set of non-sensitive words. Based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words, it is determined whether the text to be detected contains sensitive words.

2. The text detection method as described in claim 1, characterized in that, The step of determining whether the text to be detected contains sensitive words based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words includes: Calculate the first average confidence level of all sensitive words in the first set of sensitive words; Calculate the second average confidence level of all non-sensitive words in the first set of non-sensitive words; Calculate the third average confidence level of all sensitive words in the second set of sensitive words; Calculate the fourth average confidence level of all non-sensitive words in the second set of non-sensitive words; Based on the first average confidence level, the second average confidence level, the third average confidence level, and the fourth average confidence level, it is determined whether the text to be detected contains the sensitive word.

3. The text detection method as described in claim 2, characterized in that, The step of determining whether the text to be detected contains the sensitive word based on the first average confidence level, the second average confidence level, the third average confidence level, and the fourth average confidence level includes: A first fused confidence level is obtained by fusing the first average confidence level and the third average confidence level. A second fused confidence level is obtained by fusing the second average confidence level and the fourth average confidence level. Based on the first fusion confidence score and the second fusion confidence score, it is determined whether the text to be detected contains the sensitive word.

4. The text detection method as described in claim 3, characterized in that, The step of determining whether the text to be detected contains the sensitive word based on the first fusion confidence score and the second fusion confidence score includes: Based on the first fusion confidence score and the second fusion confidence score, the target label confidence score of the text to be detected is determined; Based on the confidence level of the target label, it is determined whether the text to be detected contains the sensitive word.

5. The text detection method as described in claim 4, characterized in that, The step of determining the target label confidence of the text to be detected based on the first fusion confidence and the second fusion confidence includes: When the first fusion confidence score is greater than the second fusion confidence score, the first fusion confidence score is determined as the target label confidence score; When the first fusion confidence score is less than the second fusion confidence score, the second fusion confidence score is determined as the target label confidence score; When the first fusion confidence level is equal to the second fusion confidence level, the target label confidence level is empty.

6. The text detection method as described in claim 4, characterized in that, The step of determining whether the text to be detected contains the sensitive word based on the confidence score of the target label includes: When the confidence level of the target label is greater than the preset confidence level threshold, it is determined whether the text to be detected contains the sensitive word based on the confidence level of the target label. When the confidence level of the target label is not greater than the preset confidence level threshold or the confidence level of the target label is empty, the text to be detected is fed back to the relevant personnel corresponding to the text to be detected, so that the relevant personnel can review the text to be detected and determine whether the text to be detected contains the sensitive words.

7. The text detection method as described in claim 6, characterized in that, After the method involves sending the text to be detected to the relevant personnel so that they can review it and determine whether it contains the sensitive word, the method further includes: When the text to be detected contains the sensitive word, the target sensitive word is extracted from the text to be detected; Based on the target sensitive words, the preset sensitive word tree is updated.

8. A text detection device, characterized in that, The text detection device includes: The first acquisition module is used to acquire the text to be detected; The first detection module is used to perform sensitive word detection processing on the text to be detected based on a preset sensitive word tree, so as to obtain a first set of sensitive words and a first set of non-sensitive words; The second detection module is used to perform sensitive word detection processing on the text to be detected based on a preset large language model, so as to obtain a second set of sensitive words and a second set of non-sensitive words. The first determining module is used to determine whether the text to be detected contains sensitive words based on the first set of sensitive words, the first set of non-sensitive words, the second set of sensitive words, and the second set of non-sensitive words.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the text detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the text detection method as described in any one of claims 1 to 7.

Images