Text security judgment method and device based on consensus verification

Abstract

The application provides a text security judgment method and device based on consensus verification, and belongs to the technical field of natural language processing. The method comprises the following steps: obtaining a target text; inputting the target text into a judgment model to obtain a judgment result output by the judgment model, wherein the judgment result is used to indicate whether the target text is a safe text, the safe text does not include harmful expressions, the harmful expressions include a first harmful new word, and the first harmful new word is a word with harmful word meaning in the target text; wherein the first harmful new word is determined based on consensus verification, and the consensus verification is based on information searched by a search engine. The harmful new word with different word meanings in the context and the independent use can be accurately identified, the identification ability of implicit harmful network expressions is enhanced, and the accuracy and comprehensiveness of network text harmful content detection are improved.

Description

Technical Field

[0001] This application belongs to the field of natural language processing technology, specifically relating to a text security judgment method and apparatus based on consensus verification. Background Technology

[0002] With the continuous development of the internet social ecosystem, online language has shown characteristics of rapid evolution and segmented dissemination. Various new words and memes continue to emerge and are widely used. Some seemingly neutral expressions are given implicit harmful meanings in the interaction and dissemination of specific communities, becoming hidden carriers for the spread of harmful content.

[0003] Current technologies for detecting harmful content in online texts largely rely on static models. The detection capabilities of these models are limited by the time frame and content coverage of the training data, making it difficult to capture the dynamic evolution of online language. For new words and trending terms that lack obvious harmful characteristics but possess harmful meanings only in specific contexts, they cannot accurately identify the harmful information or intent they express. Furthermore, existing detection technologies lack the ability to effectively mine semantic consensus within the public online context, failing to rely on massive amounts of external online information to determine the actual semantic meaning of words, and instead relying solely on the model's own static knowledge reserves. This leads to a high likelihood of missed detections when dealing with words whose semantics have shifted, making it difficult to meet the comprehensive and accurate requirements for harmful content detection in cyberspace content security governance. Summary of the Invention

[0004] This application provides a consensus-based text security judgment method and apparatus to accurately identify whether there are harmful new words in the target text, enhance the ability to identify implicit harmful online expressions, and improve the accuracy and comprehensiveness of detecting harmful content in online text.

[0005] This application provides a text security judgment method based on consensus verification, including: Get the target text; The target text is input into the judgment model to obtain the judgment result output by the judgment model. The judgment result is used to indicate whether the target text is safe text. The safe text does not include harmful expressions. The harmful expressions include the presence of a first harmful new word. The first harmful new word is a word in the target text whose meaning is harmful. The first harmful new word is determined based on consensus verification, which is based on information retrieved by a search engine.

[0006] According to the consensus-based text security judgment method provided in this application, the reasoning process of the judgment model includes: determining whether the target text includes a second harmful new word, wherein the meaning of the second harmful new word includes a harmful meaning when used independently; if the second harmful new word is included, determining the first true intent of the second harmful new word in the target text based on the second harmful new word; and judging whether the target text is a safe text based on the first true intent.

[0007] According to the consensus-based text security judgment method provided in this application, the method further includes: if the target text is determined to be safe text based on the first true intent, determining whether the target text includes the first harmful new word; if the target text includes the first harmful new word, determining whether the target text is safe text based on the first harmful new word.

[0008] According to the consensus-based text security judgment method provided in this application, determining whether the target text includes the first harmful new word includes: locating candidate words from the target sentence; calling a search engine to search for the candidate words and obtaining search results; judging the harmfulness of the candidate words based on the search results and obtaining a judgment result; and determining whether the candidate words are the first harmful new word based on the judgment result.

[0009] According to the consensus-based text security judgment method provided in this application, the step of judging the harmfulness of the candidate words based on the search results to obtain a judgment result includes: determining at least one word description for the candidate words based on the search results; substituting the at least one word description into the target text to obtain at least one second true intent of the candidate words in the target text; and judging the harmfulness of the candidate words based on the at least one second true intent to obtain a judgment result.

[0010] According to the consensus-based text security judgment method provided in this application, the step of substituting the at least one word description into the target text to obtain at least one second true intent of the candidate word in the target text includes: obtaining the context of the target text; analyzing the context to obtain context semantics; and determining the context semantics and the at least one word description to obtain at least one second true intent of the candidate word in the target text.

[0011] According to the consensus-based text security judgment method provided in this application, the judgment result includes at least one of the following: a security label of the target text, wherein the security label is used to indicate whether the target text is harmful; and the reasoning process for whether the target text is harmful.

[0012] According to the consensus-based text security judgment method provided in this application, the judgment model is trained in the following manner: harmful new word samples are obtained from a harmful new word library; general samples are obtained from a general sample library, wherein the general samples include words whose meanings are the same when used independently and when used in sentences; a hybrid dataset is obtained based on the harmful new word samples and the general samples; and the base model is supervised and fine-tuned based on the hybrid dataset to obtain the judgment model.

[0013] According to the consensus-based text security judgment method provided in this application, the method further includes: obtaining multiple publicly available texts from a public platform; extracting reference words from the multiple publicly available texts using a general large model based on preset prompt words, wherein the preset prompt words include prompt text and a first harmful new word included in the prompt text; using the reference words as query terms to search in a search engine to obtain search results; performing consensus verification on the reference words using a general large model based on the search results to obtain consensus verification results; determining target words from the reference words based on the consensus verification results; and constructing the harmful new word database based on the target words.

[0014] According to the consensus-based text security judgment method provided in this application, the step of constructing the harmful new word library based on the target words includes: annotating the target words according to the search results to obtain annotated content; verifying the annotated content to obtain a verification result; and adding the target words that have passed the verification indicated by the verification result to the harmful new word library.

[0015] According to the consensus-based text security judgment method provided in this application, the annotation content includes at least one of the following: the original semantics of the target word; the current semantics of the target word; the evolution source of the current semantics; and an explanation of the evolution from the original semantics to the current semantics.

[0016] This application also provides a text security judgment device based on consensus verification, including: The acquisition unit is used to acquire the target text; The judgment unit is used to input the target text into the judgment model and obtain the judgment result output by the judgment model. The judgment result is used to indicate whether the target text is safe text. The safe text does not include harmful expressions. The harmful expressions include the presence of harmful new words. The harmful new words are words in the target text whose meaning is harmful. The harmful new words are determined based on consensus verification, which is based on information retrieved by a search engine.

[0017] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the text security judgment method based on consensus verification as described above.

[0018] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the text security judgment method based on consensus verification as described above.

[0019] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the consensus-based text security judgment method as described above.

[0020] The consensus-based text security assessment method and apparatus provided in this application first acquire target text; input the target text into a assessment model to obtain the assessment result output by the assessment model. The assessment result indicates whether the target text is safe text, wherein the safe text does not contain harmful expressions, and the harmful expressions include the presence of a first harmful new word, which is a word in the target text whose meaning is harmful. The first harmful new word is determined based on consensus verification, which is based on information retrieved by a search engine. This solution determines the presence of a first harmful new word through consensus verification based on search engine information, accurately identifying first harmful new words used independently with meanings different from their context. This effectively enhances the ability to identify implicitly harmful online expressions and significantly improves the accuracy and comprehensiveness of detecting harmful content in online text. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the consensus-based text security assessment method provided in this application.

[0023] Figure 2 This is a schematic diagram of the data construction and identification framework provided in this application.

[0024] Figure 3 This is a block diagram of the functional units of the consensus-based text security judgment device provided in this application.

[0025] Figure 4This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0029] Current technologies for detecting harmful content in online texts rely on static models, which are limited by training data and struggle to capture the dynamic evolution of online language. They are unable to accurately identify the harmful meanings of new words and trending terms in specific contexts. Furthermore, due to a lack of ability to mine semantic consensus within the public online context and an inability to determine the actual semantic meaning of words based on external online information, they are prone to missing detections of semantically transferred words, failing to meet the comprehensive and accurate requirements of online content security governance for harmful content detection.

[0030] To address the aforementioned problems, this application provides a text security assessment method and apparatus based on consensus verification. The following detailed description, in conjunction with specific embodiments, further illustrates this application.

[0031] Please see Figure 1 , Figure 1 This is a flowchart illustrating the consensus-based text security judgment method provided in this application. The consensus-based harmful new word judgment method includes the following steps.

[0032] S101, Obtain the target text.

[0033] The target text refers to online text statements to be assessed for security. These statements can originate from various online social platforms, public online publishing channels, and online information dissemination carriers, and may contain text content that uses trending words and memes as a vehicle for implicitly harmful expressions. They can be independent single statements or excerpts from continuous text, fully representing the usage of words in a specific context.

[0034] S102, input the target text into the judgment model to obtain the judgment result output by the judgment model.

[0035] The determination result indicates whether the target text is safe text. Safe text does not contain harmful expressions. Harmful expressions include the presence of a first harmful new word, which is a word in the target text with a harmful meaning. The first harmful new word is determined based on consensus verification, which is based on information retrieved by a search engine. In practice, the target text input to the judgment model in this step can be directly input as text data. The judgment model can perform semantic analysis and feature extraction on the target text to identify and judge whether there are harmful expressions in the target text, and output the corresponding judgment result. This judgment result can directly and intuitively reflect the presence of harmful expressions in the target text, providing a direct basis for the review of harmful content in online texts. The first harmful neologism can specifically include various new words and memes generated and spread in online communities and platform ecosystems, whose semantics have been solidified through group interaction. These can be existing words that have undergone semantic reassignment, or newly created words and combined expressions. These words can be given harmful semantics such as illegal behavior, extreme stance, inflammatory narrative, and illegal or prohibited content in specific community contexts. When used independently, they present a safe original meaning, but in the specific context of the target text, they are given harmful semantics different from the original meaning.

[0036] As can be seen, this solution uses harmful new word samples determined by consensus verification based on search engine information to train the judgment model, and then inputs the target text into the judgment model for detection. This can accurately identify the first harmful new word used independently with a different meaning from the word in the context, effectively enhancing the ability to identify implicit harmful online expressions and significantly improving the accuracy and comprehensiveness of harmful content detection in online text.

[0037] In one possible embodiment, the reasoning process of the judgment model includes: determining whether the target text contains a second harmful new word, the meaning of which includes a harmful meaning when used independently; if the second harmful new word is included, determining the first true intent of the second harmful new word in the target text based on the second harmful new word; and determining whether the target text is a safe text based on the first true intent.

[0038] Among them, the second harmful neologism is a word whose meaning is inherently harmful, or whose harmful meaning has been widely used to express harmful content. In this case, the judgment model can determine that the target text may contain harmful expressions without consensus verification. The intent of the second harmful neologism in the target text, as well as its meaning, can then be used to determine whether the target text is safe.

[0039] In one possible embodiment, the method further includes: if the target text is determined to be safe text based on the first true intent, determining whether the target text includes the first harmful new word; if the target text includes the first harmful new word, determining whether the target text is safe text based on the first harmful new word.

[0040] In determining whether a target text is safe, the system first checks whether a second harmful new word exists. If a second harmful new word exists, a safety assessment is performed based on that word. If the text is deemed safe based on the second harmful new word, then the system checks whether a first harmful new word exists. If a first harmful new word exists, a safety assessment is performed based on that word.

[0041] Since the second harmful new word is widely identified as a word used for harmful expressions, the safety of the target text is first judged based on the second harmful new word, and then the safety of the target text is judged based on the first harmful word verified by consensus based on the search results. This can improve both the efficiency and accuracy of the judgment.

[0042] In one possible embodiment, determining whether the target text includes the first harmful new word includes: locating candidate words from the target sentence; calling a search engine to search for the candidate words and obtaining search results; judging the harmfulness of the candidate words based on the search results and obtaining a judgment result; and determining whether the candidate words are the first harmful new word based on the judgment result.

[0043] The process for locating candidate words involves first performing a complete semantic analysis and lexical segmentation of the target text, then filtering out words that are semantically neutral, exhibit semantic ambiguity, or belong to the category of trending internet slang or memes as candidate words. It can also incorporate a harmful new word database to match words from the target text that conform to emerging expressions as candidate words. Simultaneously, it can independently examine the overall semantics of the target text and the behavior and intent it conveys, thereby identifying words that may be given special semantic meaning within the sentence, ensuring that the located candidate words cover all words in the target sentence that may exhibit semantic shift.

[0044] When using a search engine, candidate words obtained from the search can be directly used as query terms. This triggers the search engine to collect, organize, and process information from massive amounts of data on the internet. The final search results can include publicly available information from the first page returned by the search engine. They can also cover online semantic explanations related to the candidate words, actual usage scenarios, social media contexts, and semantic descriptions of the words from different information sources.

[0045] The harmfulness assessment method for candidate words can leverage the semantic understanding capabilities of large language models to perform in-depth analysis and summarization of search results, extracting the mainstream semantic connotations of candidate words in the network, and inferring the true usage intent of candidate words in conjunction with the specific context of the target sentence. Simultaneously, it can determine whether the semantics of candidate words in the target text point to harmful information or intent, forming a final harmfulness assessment result through multi-dimensional analysis and reasoning. The assessment model can directly determine whether a candidate word is a second harmful neologism used independently with a different meaning from its context based on this harmfulness assessment result. The entire reasoning process can be automatically completed by the algorithmic logic of the assessment model, or the cross-task generality of large language models can be combined to improve the accuracy of the reasoning, enabling the assessment model to accurately identify harmful neologisms in the target sentence that use trending terms and memes as carriers.

[0046] As can be seen, this solution first locates candidate words in the target sentence during the reasoning process, then calls the search engine to obtain the search results for the candidate words and makes a harmfulness judgment based on them, ultimately determining whether the candidate words are harmful new words. It can rely on real-time network information from the search engine to mine the actual semantic meaning of candidate words, accurately determine the true intent of candidate words in specific contexts, effectively enhance the accuracy of identifying second-generation harmful new words based on semantic transfer, and significantly reduce the probability of missed detection of harmful content in online text.

[0047] In one possible embodiment, the step of determining the harmfulness of the candidate words based on the search results and obtaining a determination result includes: determining at least one word description for the candidate words based on the search results; substituting the at least one word description into the target text to obtain at least one second true intent of the candidate words in the target text; and determining the harmfulness of the candidate words based on the at least one second true intent to obtain a determination result.

[0048] In the process of determining the harmfulness of candidate words, a large language model can be used to perform in-depth semantic extraction and summarization of the search results returned by the search engine. This allows for the screening of the mainstream semantic orientation of candidate words in the public online context from a vast amount of publicly available information, thereby determining at least one semantic description for each candidate word. This semantic description can cover the original neutral semantics of the candidate word, as well as various derived semantics assigned to it in the online community, especially potential harmful semantics such as attacks, illegal or prohibited content. It can fully present all semantic possibilities of the candidate word in actual online dissemination.

[0049] As can be seen, this solution effectively improves the accuracy of judging the security of target text by determining the semantic description of candidate words from search results and substituting them into the target text to obtain the true intent of the candidate words, and then judging the security of the target text based on the true intent, thus avoiding the misjudgment problem caused by single semantic judgment.

[0050] In one possible embodiment, the step of substituting the at least one word description into the target text to obtain at least one second true intent of the candidate word in the target text includes: obtaining the context of the target text; analyzing the context to obtain context semantics; and determining the context semantics and the at least one word description to obtain at least one second true intent of the candidate word in the target text.

[0051] The acquired context can be the continuous text content of the target text, the related contextual information in the target text's dissemination scenario, or relevant content such as interactive comments and topic backgrounds. It can fully present the actual usage context of the target text, providing comprehensive contextual support for subsequent contextual semantic analysis. After obtaining the context of the target text, the semantic understanding and contextual analysis capabilities of the large language model can be used to perform a holistic analysis of the context. During the analysis, the layered expression characteristics and metaphorical usage logic of online language can be combined to uncover the overall semantic tendency and specific expressive intent conveyed by the context, accurately obtaining the contextual semantics that reflects the actual usage scenario of the target statement. This analysis process can also be combined with a contextual feature database of online text to improve the accuracy of analyzing special contexts and layered contexts.

[0052] After completing the contextual semantic analysis, the contextual semantics can be compared with each second true intent. The intent consistent with the contextual semantics among the second true intents is determined as the final intent. Then, based on the final intent, it is determined whether the target text is safe text. That is, for example, if it is determined that there are safe intents and insecure intents among the second true intents, but the determined contextual semantics are insecure semantics, then the second true intent expressing the insecure intent can be determined as the final intent of the target text.

[0053] As can be seen, this solution further acquires the context of the target text and analyzes its semantics, comprehensively judging the safety of the target text based on the contextual semantics. It can cover new words and trending phrases that have not yet formed a public consensus on harmful semantics but convey harmful semantics in specific contexts, effectively making up for the limitations of relying solely on quantity judgment, enhancing the ability to identify implicitly harmful new words in circle-based and contextualized contexts, significantly improving the comprehensiveness of harmful new word detection, and reducing the probability of false negatives.

[0054] In one possible embodiment, the determination result includes at least one of the following: a security label of the target text, the security label being used to indicate whether the target text is harmful; and a reasoning process for determining whether the target text is harmful.

[0055] This solution can output the judgment labels for candidate words separately, output the labels and reasoning process simultaneously, or output only the reasoning process for reviewers to refer to, thus adapting to the display needs of judgment results in different review scenarios. The safety labels used to indicate whether the target text is harmful can be set to a standardized identifier format, which can be intuitive text labels such as "safe" or "unsafe," or preset number or symbol labels. The judgment results reflected by the labels are all derived by the judgment model based on the complete intent judgment process of the target text, and can directly and clearly indicate whether the target text contains harmful expressions.

[0056] The reasoning process for determining whether target text is harmful is a complete logical outline and presentation of the judgment model's harmfulness assessment. This reasoning process can fully reconstruct all steps of the judgment model, from locating harmful new words, calling the search engine to obtain search results, to combining the search results to determine intent and arrive at the final conclusion. It can also present the core judgment criteria in each step in detail, including the second harmful new word obtained, the first true intent based on the second harmful new word, the located candidate words, the semantic information of the candidate words obtained from the search engine, and the second true intent based on the first harmful new word. If the context-based fallback judgment logic is triggered, the acquisition and analysis results of the context can also be presented during the reasoning process, ensuring that the reasoning process completely corresponds to the actual judgment logic of the judgment model.

[0057] In actual network content security review processes, this reasoning process can be output in structured text form, clearly explaining the harmful meaning of seemingly neutral candidate words under the current network consensus and the basis for judgment. The complete reasoning process can provide content reviewers with reliable decision-making basis. At the same time, the diverse presentation formats of the judgment results allow this solution's text security judgment method to meet both the need for rapid judgment results in automated review scenarios and the need for traceability of judgment basis and judgment logic in manual review scenarios. This makes the entire judgment process interpretable and avoids the review decision-making difficulties caused by traditional detection models that only output black-box results.

[0058] In one possible embodiment, the judgment model is trained as follows: harmful new word samples are obtained from a harmful new word library; general samples are obtained from a general sample library, the general samples including words whose meanings are the same when used independently and when used in sentences; a hybrid dataset is obtained based on the harmful new word samples and the general samples; and the base model is supervised and fine-tuned based on the hybrid dataset to obtain the judgment model.

[0059] The base model's inference framework can be the SeTox framework, which is built upon the collaborative interaction logic between an external search engine and a large language model. Its core revolves around a holistic inference system constructed from search terms validated through consensus based on search engine information. The framework allows for pre-defined structured inference step guidance, enabling the model to master the complete logic of identifying candidate options, invoking the search, and making comprehensive inference judgments. Furthermore, this framework is adaptable to base models of different architectures and scales, and can optimize model capabilities through lightweight inference-guided training without requiring full parameter retraining.

[0060] The general sample text can include routine sentences used in scenarios such as daily online communication, formal media publications, and general written expression. The words in these sentences maintain their independent usage and consistent meaning within the sentence, without semantic transfer or implicit harmful semantics. It can encompass both neutral general text without any harmful information and general text containing explicit harmful expressions, enabling the judgment model to distinguish between routine semantic expressions and harmful expressions involving semantic transfer during training, thus strengthening the model's basic semantic judgment capabilities.

[0061] After acquiring harmful new word samples and general samples, the two types of samples can be mixed and integrated according to a preset ratio to obtain a hybrid dataset. The mixing ratio can be flexibly adjusted according to the actual model training needs and detection scenarios. It can either increase the proportion of harmful new word samples to enhance the model's ability to identify harmful new words, or combine them with a reasonable proportion of general samples to ensure the model's generalization. The hybrid dataset can be standardized in a unified format so that the samples can be adapted to the training input requirements of the base model. Supervised fine-tuning of the base model includes supervised fine-tuning training based on the hybrid dataset. During the training process, the model can be guided to learn the recognition logic and reasoning methods for harmful new words, enabling the model to generate structured reasoning chains when processing new words. At the same time, the model can learn to combine external search results with sentence context to determine harmfulness through labeled samples. During the fine-tuning process, targeted training can also be performed on the model's reasoning steps to enhance the model's ability to locate candidate words, call search engines to obtain semantics, and combine semantic analysis intent. It can also be fine-tuned and optimized in multiple rounds based on the training effect of the model. In each round, the training parameters and sample input can be adjusted according to the recognition accuracy of the model, so that the trained judgment model can accurately identify harmful new words in the target sentence. At the same time, this supervised fine-tuning method can rely on the characteristics of the SeTox framework to achieve adaptive training for different base models, so that the trained judgment model has the core ability to dynamically retrieve external information and combine it with context analysis, thus meeting the actual needs of detecting harmful expressions of new internet slang and memes.

[0062] In one possible embodiment, the method further includes: obtaining multiple publicly available texts from a public platform; extracting reference words from the multiple publicly available texts using a general large model based on preset prompt words, wherein the preset prompt words include prompt text and a first harmful new word included in the prompt text; using the reference words as query terms to search in a search engine to obtain search results; performing consensus verification on the reference words using a general large model based on the search results to obtain consensus verification results; determining target words from the reference words according to the consensus verification results; and constructing the harmful new word database according to the target words.

[0063] The publicly available statements obtained can come from online public platforms, social media platforms, or be simultaneously obtained from existing online harmful expression data resources. When extracting reference words, preset prompts can be set based on the characteristics of harmful new words and the evolutionary patterns of online language. The prompt text can clearly define the extraction requirements, and the prompt words can be selected as typical first-time harmful new words as reference examples.

[0064] When a reference word is used as a query term in a search engine, the search engine can collect, organize, and process relevant information about the reference word from massive amounts of data on the Internet. The final search results can include publicly available information on the search engine's homepage, online semantic explanations of the reference word, actual usage scenarios, semantic descriptions of the word from different information sources, and the context of its dissemination in social media.

[0065] Consensus verification based on search results can utilize a general-purpose large language model as the consensus verifier. This model can perform in-depth semantic analysis and summarization of the search results, leveraging the strong semantic understanding capabilities of the general-purpose large language model to accurately uncover public semantic consensus within the search results, ensuring that the judgment aligns with the actual dissemination characteristics of online language. After consensus verification is completed, reference words verified as having formed a harmful semantic consensus at the public level can be identified as target words. These target words are all new words and trending terms that are semantically neutral when used independently but are given harmful semantics in specific contexts, enabling precise matching of the core features of harmful new words.

[0066] Meanwhile, the construction of a harmful new word database can also include a dynamic update process. New target words can be continuously collected and labeled according to the evolution of online language, constantly enriching the content of the database and ensuring that the harmful new word database can keep up with the evolutionary characteristics of new online words and memes in real time.

[0067] As can be seen, this solution obtains public statements from multiple sources and extracts reference words, combines them with search engine retrieval to complete consensus verification, and then determines target words and constructs a harmful new word library. It can dynamically mine the first harmful new words based on real-time network public semantic consensus, ensuring the timeliness and accuracy of the harmful new word library content, providing high-quality harmful new word samples for the judgment model training, and enhancing the judgment model's ability to identify various emerging harmful new words and memes from the data source.

[0068] In one possible embodiment, constructing the harmful new word library based on the target words includes: tagging the target words according to the search results to obtain tagged content; verifying the tagged content to obtain a verification result; and adding the target words that passed the verification to the harmful new word library.

[0069] The verification methods can include a two-round verification combined with multi-dimensional cross-judgment. Multiple groups of reviewers can conduct independent verification, and then cross-check the results. Simultaneously, the labeled content is reviewed and judged in conjunction with the actual semantic consensus of the network to ensure the accuracy and completeness of the labeled information.

[0070] In one possible embodiment, the annotation content includes at least one of the following: the original semantics of the target word; the current semantics of the target word; the source of the evolution of the current semantics; and a description of the evolution from the original semantics to the current semantics.

[0071] When annotating target words, all annotation content is based primarily on the search results returned by the search engine for the target word. The inherent dictionary meaning and common usage meaning of the target word can be extracted from the search results as the original semantic annotation, ensuring consistency with the neutral meaning when the word is used independently. Simultaneously, harmful semantics formed by the target word in the online public context and solidified through group interaction and repeated use are summarized from the search results as current semantic annotations. The current semantics can clearly point to specific risk categories. Furthermore, information such as the initial propagation scenarios and triggering reasons for the harmful semantics of the target word can be mined from the search results as evolution source annotations. The propagation path, solidification process, and usage circles of harmful semantics in the network can be traced as evolution explanation annotations. The entire annotation process can be completed according to a standardized term format, ensuring that the annotation content of each target word forms complete and standardized metadata information, clearly presenting the semantic evolution process and harmful semantic indications of the target word.

[0072] In practice, the first round of verification focuses on checking the completeness of the labeled content and the rationality of its semantic meaning. The second round of verification reviews and corrects the results of the first round. Simultaneously, multi-dimensional cross-judgment allows for cross-verification of the labeled content with the search engine's original search results. Judgment can also be based on actual online usage examples of the target term. Furthermore, a comprehensive judgment can be made by referring to semantic descriptions of the target term from different information sources.

[0073] If any discrepancies, missing information, or labeling errors are found between the labeled content and the actual semantic consensus of the network during the verification process, the labeled content can be corrected in a timely manner. After correction, the verification process can be restarted until the labeled content meets the verification standards. The final verification result can clearly indicate whether the labeled content of each target word has passed the verification.

[0074] After completing all verification work, the target words that passed the verification can be systematically categorized and organized according to their harmful semantic categories. Then, the organized target words and their complete annotations can be added to the new harmful term library. During the addition process, the term information in the new harmful term library can be updated synchronously. Target words that failed verification can be temporarily left unadded to the new harmful term library until their annotations are corrected and they pass verification again, at which point they will be included.

[0075] As can be seen, this solution standardizes the target words by performing multi-dimensional annotations, including original semantics and current semantics, based on the search results. After verifying the annotation content, qualified target words are added to the harmful new word database. This ensures the accuracy, completeness, and authority of the word information in the harmful new word database, allowing it to accurately reflect the semantic features and evolution of harmful new words and memes on the internet. This provides high-quality sample data for training the judgment model and effectively improves the accuracy of the judgment model in identifying harmful new words.

[0076] Please refer to the following: Figure 2 This document will introduce the overall process of this application.

[0077] Starting with the input text, the process first collects various text data from public platforms, social media, and existing resources. This includes both conventional expressions and harmful expressions with trending new words and memes, providing a foundation for subsequent candidate word mining. Then, suspected harmful candidate words are extracted from the collected text to form a candidate word pool. These candidate words are used as query terms in a search engine to retrieve corresponding search results from the public internet. Based on the search results, a public consensus verification process is then conducted. Words deemed harmful enter the structured metadata annotation stage, undergoing two rounds of rigorous manual verification. The final harmful new word database, containing verified harmful new words, is constructed to provide high-quality harmful new word samples for model training.

[0078] Based on a hybrid dataset formed by constructing a new harmful lexicon and general samples, the base model is trained to master the core reasoning logic of the SeTox framework. After training, the model enters the core SeTox security reasoning generation stage. For the input text to be detected, candidate nouns are first located and a search tool is invoked. Then, the real-time semantic information of the candidate nouns is obtained using a search engine. Combined with text analysis, the true meaning of the words in the specific context is analyzed, and finally, a label indicating whether the text is safe is output. Simultaneously, reasoning content containing the judgment criteria is generated. For example, for the input text "I don't want to talk to someone like you," the model will label <noun: xx> and trigger a search. Combining the retrieved public consensus, the harmful semantics of "xx" are analyzed, and the final label "unsafe" is output. The corresponding reasoning analysis content is provided simultaneously, fully realizing the entire execution logic from data collection and lexicon construction to the final output of a security judgment result with reasoning process for the text to be detected.

[0079] The following describes a consensus-based text security assessment device provided in this application. The consensus-based text security assessment device described below corresponds to the consensus-based text security assessment method described above.

[0080] Please see Figure 3 The consensus-based text security assessment device 300 includes: an acquisition unit 301 for acquiring target text; and a assessment unit 302 for inputting the target text into an assessment model and obtaining an assessment result output by the assessment model. The assessment result indicates whether the target text is safe text, wherein the safe text does not contain harmful expressions, and the harmful expressions include the presence of harmful new words, which are words in the target text whose meaning is harmful. The harmful new words are determined based on consensus verification, which is based on information retrieved by a search engine.

[0081] In one possible embodiment, the judgment model includes a reasoning unit, which is specifically used to: determine whether the target text includes a second harmful new word, the meaning of which includes a harmful meaning when used independently; if the second harmful new word is included, determine the first true intent of the second harmful new word in the target text based on the second harmful new word; and determine whether the target text is safe text based on the first true intent.

[0082] In one possible embodiment, the reasoning unit is further configured to: determine whether the target text includes the first harmful new word if the target text is determined to be safe text based on the first true intent; if the target text includes the first harmful new word, determine whether the target text is safe text based on the first harmful new word.

[0083] In one possible embodiment, in determining whether the target text includes the first harmful new word, the reasoning unit is specifically configured to: locate candidate words from the target sentence; call a search engine to search for the candidate words and obtain search results; perform a harmfulness judgment on the candidate words based on the search results and obtain a judgment result; and determine whether the candidate words are the first harmful new word based on the judgment result.

[0084] In one possible embodiment, in the step of determining the harmfulness of the candidate words based on the search results and obtaining a determination result, the reasoning unit is specifically configured to: determine at least one word description for the candidate words based on the search results; substitute the at least one word description into the target text to obtain at least one second true intent of the candidate words in the target text; and determine the harmfulness of the candidate words based on the at least one second true intent to obtain a determination result.

[0085] In one possible embodiment, in order to obtain at least one second true intent of the candidate word in the target text by respectively incorporating the at least one word description into the target text, the reasoning unit is specifically configured to: obtain the context of the target text; analyze the context to obtain context semantics; determine the context semantics and the at least one word description to obtain at least one second true intent of the candidate word in the target text.

[0086] In one possible embodiment, the determination result includes at least one of the following: a security label of the target text, the security label being used to indicate whether the target text is harmful; and a reasoning process for determining whether the target text is harmful.

[0087] In one possible embodiment, the consensus-based text security judgment device 300 further includes a training unit, which is specifically used for: obtaining harmful new word samples from a harmful new word library; obtaining general samples from a general sample library, wherein the general samples include words whose meanings are the same when used independently and when used in sentences; obtaining a hybrid dataset based on the harmful new word samples and the general samples; and performing supervised fine-tuning of the base model based on the hybrid dataset to obtain the judgment model.

[0088] In one possible embodiment, the consensus-based text security judgment device 300 further includes a construction unit, which is specifically used for: obtaining multiple public texts from a public platform; extracting reference words from the multiple public texts based on preset prompt words using a general large model, wherein the preset prompt words include prompt text and a first harmful new word included in the prompt text; using the reference words as query terms to search in a search engine to obtain search results; performing consensus verification on the reference words using a general large model based on the search results to obtain consensus verification results; determining target words from the reference words according to the consensus verification results; and constructing the harmful new word database according to the target words.

[0089] In one possible embodiment, in constructing the harmful new word library based on the target words, the construction unit is specifically used to: annotate the target words according to the search results to obtain annotated content; verify the annotated content to obtain a verification result; and add the target words that have passed the verification to the harmful new word library.

[0090] In one possible embodiment, the annotation content includes at least one of the following: the original semantics of the target word; the current semantics of the target word; the source of the evolution of the current semantics; and a description of the evolution from the original semantics to the current semantics.

[0091] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. For example... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a consensus-based text security judgment method. This method includes: acquiring target text; inputting the target text into a judgment model to obtain a judgment result output by the judgment model, the judgment result indicating whether the target text is safe text, wherein the safe text does not contain harmful expressions, and the harmful expressions include the presence of a first harmful new word, the first harmful new word being a word in the target text with a harmful meaning; wherein the first harmful new word is determined based on consensus verification, and the consensus verification is based on information retrieved by a search engine.

[0092] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a consensus-based text security judgment method provided by the above methods. The method includes: acquiring target text; inputting the target text into a judgment model to obtain a judgment result output by the judgment model, the judgment result indicating whether the target text is safe text, wherein the safe text does not contain harmful expressions, the harmful expressions including the presence of a first harmful new word, the first harmful new word being a word in the target text with a harmful meaning; wherein the first harmful new word is determined based on consensus verification, the consensus verification being based on information retrieved by a search engine.

[0094] In another aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described consensus-based text security judgment methods. The method includes: acquiring target text; inputting the target text into a judgment model to obtain a judgment result output by the judgment model, the judgment result indicating whether the target text is safe text, wherein the safe text does not contain harmful expressions, the harmful expressions including the presence of a first harmful new word, the first harmful new word being a word in the target text with a harmful meaning; wherein the first harmful new word is determined based on consensus verification, the consensus verification being based on information retrieved by a search engine.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text security judgment method based on consensus verification, characterized in that, include: Get the target text; The target text is input into the judgment model to obtain the judgment result output by the judgment model. The judgment result is used to indicate whether the target text is safe text. The safe text does not include harmful expressions. The harmful expressions include the presence of a first harmful new word. The first harmful new word is a word in the target text whose meaning is harmful. The first harmful new word is determined based on consensus verification, which is based on information retrieved by a search engine.

2. The method according to claim 1, characterized in that, The reasoning process of the judgment model includes: Determine whether the target text contains a second harmful new word, the meaning of which includes harmful meaning when used independently; If the second harmful new word is included, then the first true intent of the second harmful new word in the target text is determined based on the second harmful new word; Based on the first true intent, determine whether the target text is safe text.

3. The method according to claim 2, characterized in that, The method further includes: If the target text is determined to be safe text based on the first true intent, it is determined whether the target text includes the first harmful new word; If the first harmful new word is included, then the target text is determined to be safe text based on the first harmful new word.

4. The method according to claim 3, characterized in that, Determining whether the target text includes the first harmful new word includes: Locate candidate words from the target statement; The search engine is invoked to search for the candidate words, and the search results are obtained; The candidate words are judged to be harmful based on the search results, and the judgment result is obtained; Based on the judgment result, determine whether the candidate word is the first harmful new word.

5. The method according to claim 4, characterized in that, The step of determining the harmfulness of the candidate words based on the search results and obtaining the determination result includes: Determine at least one word description for the candidate words based on the search results; By substituting the at least one word description into the target text, at least one second true intent of the candidate word in the target text is obtained; Based on the at least one second true intent, the harmfulness of the candidate words is determined, and a determination result is obtained.

6. The method according to claim 5, characterized in that, The step of substituting the description of the at least one word into the target text to obtain at least one second true intent of the candidate word in the target text includes: Obtain the context of the target text; The context is analyzed to obtain contextual semantics; By determining the contextual semantics and the at least one word description, at least one second true intent of the candidate word in the target text is obtained.

7. The method according to claim 2, characterized in that, The judgment result includes at least one of the following: The security label of the target text, which is used to indicate whether the target text is harmful; The reasoning process regarding whether the target text is harmful.

8. The method according to claim 1, characterized in that, The judgment model is trained in the following way: Obtain harmful new word samples from a harmful new word database; Obtain general samples from a general sample library, which includes words whose meanings are the same when used independently and when used in sentences; A hybrid dataset is obtained based on the harmful new word samples and the general samples; The judgment model is obtained by supervising and fine-tuning the pedestal model based on the hybrid dataset.

9. The method according to claim 8, characterized in that, The method further includes: Obtain multiple publicly available texts from a public platform; Based on preset prompt words, reference words are extracted from the multiple public texts using a general large model. The preset prompt words include prompt text and a first harmful new word included in the prompt text. The reference words were used as search terms in a search engine to obtain search results. Based on the search results, a general large model is used to perform consensus verification on the reference terms to obtain consensus verification results; Target words are determined from the reference words based on the consensus verification results; The harmful new word library is constructed based on the target words.

10. The method according to claim 9, characterized in that, The construction of the harmful new word library based on the target words includes: The target words are labeled based on the search results to obtain the labeled content; The labeled content is validated, and the validation result is obtained; The verification results indicate that the target words that pass the verification are added to the harmful new word library.

11. The method according to claim 10, characterized in that, The labeled content includes at least one of the following: The original semantics of the target words; The current semantics of the target word; The source of the evolution of the current semantics; The evolution from the original semantics to the current semantics is explained.

12. A text security judgment device based on consensus verification, characterized in that, include: The acquisition unit is used to acquire the target text; The judgment unit is used to input the target text into the judgment model and obtain the judgment result output by the judgment model. The judgment result is used to indicate whether the target text is safe text. The safe text does not include harmful expressions. The harmful expressions include the presence of harmful new words. The harmful new words are words in the target text whose meaning is harmful. The harmful new words are determined based on consensus verification, which is based on information retrieved by a search engine.

Images