A domain large model content security detection method, system, device and medium

Abstract

The application relates to the technical field of large model security and discloses a domain large model content security detection method, a system, equipment and a medium, wherein the method comprises the following steps: a rule layer is constructed, the rule layer comprises a keyword library and a prompt word regular matching template library, and keywords in the keyword library and prompt word templates in the prompt word regular matching template library are respectively marked with risk levels; a model layer is constructed, the model layer comprises a sentiment polarity model and a semantic similarity model; an explanation layer is constructed, and an explanation template is constructed according to a domain large model scene requirement; the rule layer, the model layer and the explanation layer are constructed into a detector and are deployed between a user and a domain large model; content detection is performed on user requests and domain large model outputs through the rule layer and the model layer, and response information is output according to a detection result and the explanation template through the explanation layer. Through the cooperative construction of the rule layer, the model layer and the explanation layer, a multi-dimensional and full-process content security detection system is formed.

Description

Technical Field

[0001] This invention relates to the field of large model security technology, and in particular to a method, system, device and medium for detecting the security of large model content. Background Technology

[0002] With the widespread application of large-scale modeling technology across various fields, content security risks have gradually become a key factor restricting the technology's practical application. In real-world applications, improper input guidance may cause the model to generate harmful information. Issues such as false information, illegal content, and inappropriate value orientations in the generated content can seriously affect the reliability and credibility of large-scale models in various fields.

[0003] Current mainstream content security detection methods have obvious limitations: rule-based detection methods lack a deep understanding of the scenario and are prone to over-restricting normal content or missing complex violation semantics; general detection solutions fail to fully adapt to the unique needs of different domains, which may not only sacrifice the model's operating efficiency in the process of ensuring security, but also make it difficult to resist targeted malicious attacks and have weak defense capabilities when faced with flexible and ever-changing violation inducement methods.

[0004] Patent document CN120012776B discloses a content security identification method based on the integration of multiple large language models, belonging to the field of data processing technology. The method includes: receiving query text and performing word segmentation on the query text to generate query text word segments; retrieving sensitive data text matching the query text word segments from a sensitive database; concatenating the query text and sensitive data text based on a preset prompt word template to generate security identification prompt words for the query text; inputting the security identification prompt words into at least two large language models and obtaining the output results of the large language models; integrating the output results to determine the security identification result of the query text.

[0005] The aforementioned content security identification method achieves detection by integrating multiple models and combining sensitive data matching. However, it has poor adaptability to specific domain scenarios, relies on a fixed database and prompt word concatenation method, and is difficult to capture complex and ever-changing illegal semantic expressions. At the same time, the multi-model integration mode may increase computational overhead and affect the real-time response speed of detection.

[0006] Patent document CN119322978A discloses a large-scale method for detecting illegal content. The method includes: S1: inputting the user's content generation request into a Transformer model, where each layer of the Transformer model outputs a latent feature sequence, wherein the input of the current layer is the latent feature sequence output by the previous layer; S2: using an optimized spatiotemporal sequence discrimination model to analyze the violation probability of the latent feature sequences output by each layer; S3: inputting the violation probability of the latent feature sequences output by each layer into a trained secondary learner, and outputting the final violation probability.

[0007] The aforementioned large-scale illegal content detection methods rely on the extraction of potential features from each layer of the model and the output of multi-level discrimination. The model structure of such schemes is relatively complex, and the resource consumption during the inference process is high, which further exacerbates the contradiction between security assurance and operational efficiency. In addition, they lack an effective interpretation mechanism for the detection results, cannot clearly define the judgment basis, and are not conducive to subsequent optimization and adjustment. Furthermore, they are insufficient in dealing with highly adaptable illegal attack forms and cannot fully cover various illegal situations in different domain scenarios.

[0008] Therefore, addressing the personalized needs of various fields and resolving the balance between reliability, adaptability, and efficiency in large-scale model content security detection has become a crucial area for technological breakthroughs. Summary of the Invention

[0009] To address the aforementioned issues, this invention proposes a method, system, device, and medium for content security detection of a large domain model. Through the collaborative construction of the rule layer, model layer, and interpretation layer, a multi-dimensional, end-to-end content security detection system is formed.

[0010] The technical solution adopted in this invention is as follows:

[0011] A method for content security detection in a large domain model, characterized by comprising:

[0012] A rule layer is constructed, which includes a keyword library and a prompt word regular expression matching template library. Risk levels are marked for the keywords in the keyword library and the prompt word templates in the prompt word regular expression matching template library.

[0013] A model layer is constructed, which includes an emotional polarity model and a semantic similarity model. A semantic similarity judgment threshold is set, and a judgment strategy combining emotional polarity and semantic similarity is set.

[0014] Construct an interpretation layer and create interpretation templates based on the needs of the domain's large-scale model scenario;

[0015] The rule layer, model layer, and interpretation layer are constructed as detectors and deployed between the user and the domain big model. The rule layer and model layer perform content detection on user requests and domain big model outputs, and the interpretation layer outputs response information based on the detection results and interpretation templates.

[0016] Furthermore, the rule construction layer includes: constructing a keyword library through multi-source seed word collection and vocabulary enhancement and deduplication, and constructing a prompt word regular expression matching template library through risk type definition, malicious prompt sample collection, and feature extraction and transformation.

[0017] Furthermore, the rule construction layer also includes offline prompt word attack testing: collecting attack prompt words and their entity variants, conducting offline attack tests on a large domain model, extracting core words and regular expression matching templates from successfully attacked prompt words, marking risk levels and risk scenarios respectively, and adding them to the keyword library and prompt word regular expression matching template library accordingly.

[0018] Furthermore, the emotional polarity model is implemented using a lightweight bidirectional encoder representation transformation pre-trained model fine-tuned based on the training dataset.

[0019] Furthermore, the semantic similarity model adopts a combination of a bidirectional encoder representation transformation pre-trained model and a graph attention network. The bidirectional encoder representation transformation pre-trained model captures surface features, syntactic features, and semantic features. The graph attention network improves processing performance. A weighting strategy is used to weight different graph attention network layers. The model is trained using a hybrid loss function based on a keyword library.

[0020] Furthermore, content detection is performed on user requests and domain-wide model outputs through the rule layer, including: the risk level labels output by the keyword library and prompt word regular expression matching template library in the rule layer include low risk, medium risk, high risk and emergency risk. When the keyword library or prompt word regular expression matching template library outputs medium risk, high risk or emergency risk labels in real time, the answer is rejected or the output is blocked, and the violation content and matching label information are output to the interpretation layer.

[0021] Furthermore, content detection is performed on user requests and domain-wide model outputs through the model layer, including: the sentiment labels output by the sentiment polarity model in the model layer include normal neutral, positive assistance, negative inducement, and malicious attack. When the sentiment polarity label output by the sentiment polarity model in real time is negative inducement or malicious attack, and the semantic similarity judgment result of the semantic similarity model is higher than the set threshold, the answer is refused or the output is blocked, and the violation content and label information are output to the interpretation layer.

[0022] A domain-specific large model content security detection system, comprising a rule layer, a model layer, and an interpretation layer;

[0023] The rule layer includes a keyword library and a prompt word regular expression matching template library. The keywords in the keyword library and the prompt word templates in the prompt word regular expression matching template library are marked with risk levels.

[0024] The model layer includes an emotional polarity model and a semantic similarity model, and can set a semantic similarity judgment threshold and a combined judgment strategy of emotional polarity and semantic similarity.

[0025] The interpretation layer can construct interpretation templates according to the needs of a large domain model scenario.

[0026] The rule layer, model layer, and interpretation layer are deployed between the user and the domain big model in the form of detectors. The rule layer and model layer perform content detection on user requests and the output of the domain big model, and the interpretation layer outputs response information based on the detection results and interpretation templates.

[0027] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the domain large model content security detection method.

[0028] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for security detection of large-scale domain model content.

[0029] The beneficial effects of this invention are as follows:

[0030] This invention forms a multi-dimensional, full-process content security detection system through the collaborative construction of a rule layer, a model layer, and an interpretation layer. Compared with existing technologies, it has achieved many optimizations and improvements, as detailed below.

[0031] 1. This invention achieves a breakthrough in detection accuracy and balance. The rule layer uses risk level labeling based on regular expression matching of keywords and prompt words to ensure rapid identification of basic violations. The model layer deepens the understanding of complex semantics through a judgment strategy combining sentiment polarity and semantic similarity. The synergistic effect of the two avoids the excessive restriction on normal content in traditional rule detection and makes up for the omission of clear violation signals by single semantic models, effectively balancing detection accuracy and usage flexibility, and resolving the contradiction between accuracy and inclusiveness in existing solutions.

[0032] 2. This invention significantly optimizes domain adaptability and operational efficiency. The dynamic update mechanism of the static sensitive word library at the rule layer, combined with a domain-specific detection strategy, more accurately meets the unique security needs of different domains compared to existing solutions that rely on fixed databases and multi-model integration. It reduces reliance on general data, lowers unnecessary computational overhead, avoids response delays that may result from multi-model collaboration, and circumvents the excessive resource consumption caused by complex feature extraction schemes. While ensuring security detection strength, it maximizes the operational efficiency of the domain-specific large model, overcoming the efficiency loss dilemma in general detection schemes.

[0033] 3. This invention comprehensively enhances anti-attack capabilities and scenario adaptability. Through flexible setting of semantic similarity judgment thresholds and multi-dimensional analysis of sentiment polarity, this invention can more accurately identify various highly adaptable violation inducement methods. Compared with existing solutions that are insufficient in dealing with targeted attacks and flexible violation forms, it significantly improves the defense capability against malicious attacks. At the same time, the domain-customized detection logic enables the solution to quickly adapt to the security needs of different scenarios, covering more subdivided violation types, and solving the shortcomings of poor scenario adaptability and limited coverage of existing technologies.

[0034] 4. This invention offers advantages in interpretability and optimizability. The interpretation layer, built upon domain-specific scenario requirements, provides clear criteria for detection conclusions. Compared to some existing solutions that lack effective interpretation mechanisms and struggle to trace judgment logic, this not only allows users to clearly understand the origin of detection results but also provides a clear direction for subsequent iterations and optimizations of detection strategies, enhancing the maintainability and practicality of the detection system. Simultaneously, the closed-loop detection model achieves dual control over user requests and model outputs, further strengthening the comprehensiveness of content security and providing stronger support for the secure and reliable deployment of large-scale domain models. Attached Figure Description

[0035] Figure 1 This is a flowchart of a domain-wide large model content security detection method according to Embodiment 1 of the present invention.

[0036] Figure 2 This is a data flow diagram of a domain-wide large model content security detection method according to Embodiment 2 of the present invention. Detailed Implementation

[0037] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments are now described. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0038] Example 1

[0039] like Figure 1 As shown, this embodiment provides a method for content security detection of a large domain model, including:

[0040] Construct a rule layer, which includes a keyword library and a prompt word regular expression matching template library. The risk level is labeled for the keywords in the keyword library and the prompt word templates in the prompt word regular expression matching template library.

[0041] Construct a model layer, which includes an emotion polarity model and a semantic similarity model. Set a semantic similarity judgment threshold and set a judgment strategy that combines emotion polarity and semantic similarity.

[0042] Construct an interpretation layer and create interpretation templates based on the needs of the domain's large model scenarios; where scenario needs are application needs, such as user A having high needs for certain keywords or semantic security.

[0043] The rule layer, model layer, and interpretation layer are constructed as detectors and deployed between the user and the domain big model. The rule layer and model layer perform content detection on user requests and domain big model outputs, and the interpretation layer outputs response information based on the detection results and interpretation templates.

[0044] It should be noted that this method achieves full-process detection from initial screening to deep recognition through the coordinated cooperation of the rule layer, model layer and interpretation layer, which not only ensures the comprehensiveness of detection, but also improves the accuracy of detection. At the same time, the setting of the interpretation layer makes the detection results more understandable, and the deployment method of the detector does not affect the normal interaction process between the user and the large model.

[0045] Preferably, the rule layer construction includes: building a keyword library through multi-source seed word collection and vocabulary enhancement and deduplication; and building a prompt word regular expression matching template library through risk type definition, malicious prompt sample collection, and feature extraction and transformation. Specifically, when building the keyword library, initial seed words are first collected from multiple relevant channels, and then vocabulary enhancement is performed using semantic expansion, synonym expansion, and other methods. After that, a complete keyword library is formed by deduplication operations such as filtering redundant words and removing duplicate words. When building the prompt word regular expression matching template library, the various security risk types involved in the domain are first identified, and then malicious prompt samples of corresponding risk types are widely collected. Sentence structure features and structural features are extracted from the collected samples, and the extracted features are transformed into standardized regular expression matching templates, thereby constructing the prompt word regular expression matching template library.

[0046] It should be noted that multi-source collection and vocabulary enhancement ensure the coverage of the keyword library, and deduplication reduces the impact of redundant data on detection efficiency. The risk type definition provides a clear direction for malicious sample collection, and feature extraction and transformation make the regular expression matching template more targeted. The final keyword library and prompt word regular expression matching template library lay a solid foundation for efficient detection at the rule layer.

[0047] Preferably, the construction of the rule layer also includes offline prompt word attack testing: collecting attack prompt words and their entity variants, conducting offline attack tests on a large domain model, extracting core words and regular expression matching templates from the successfully attacked prompt words, marking the risk level and risk scenario respectively, and adding them to the keyword library and prompt word regular expression matching template library accordingly.

[0048] Specifically, common attack suggestion words are first collected through various public scenarios and simulations. Entity variants of these attack suggestion words are then generated based on semantic substitution and sentence transformation. Next, an offline testing environment is built, and the collected attack suggestion words and their entity variants are input into a large domain model in batches to observe and record successful attack cases. Core semantic words and key sentence structures are extracted from the successful attack suggestion words to form core vocabulary and regular expression matching templates. Then, the core vocabulary is labeled with its corresponding risk level, and the regular expression matching templates are labeled with their corresponding risk scenarios. Finally, the labeled core vocabulary is added to the keyword library, and the regular expression matching templates are added to the suggestion word regular expression matching template library.

[0049] It should be noted that offline prompt word attack testing can accurately uncover potential security risks, supplement the core vocabulary and regular expression templates for successful attacks, effectively make up for the omissions in the initial library, improve the rule layer's ability to identify various attack prompt words, reduce the risk of missed detections due to the lack of coverage of attack variants, and further enhance the comprehensiveness and reliability of the rule layer's detection.

[0050] Preferably, the sentiment polarity model is implemented using a lightweight bidirectional encoder representation transformation pre-trained model fine-tuned based on the training dataset. Specifically, text data covering various scenarios such as normal interactions and risk inducements within the domain are first collected to construct a training dataset adapted for sentiment polarity judgment. The lightweight bidirectional encoder representation transformation pre-trained model is selected as the base model, and the constructed training dataset is input into the model. By adjusting the model's hyperparameters and optimizing the training iteration process, the model is fine-tuned to enable it to accurately identify the sentiment polarity of text within the domain.

[0051] It should be noted that the lightweight model effectively reduces the computational complexity and resource consumption of the model while ensuring detection performance, making it suitable for the needs of real-time detection scenarios. The fine-tuning based on the domain-specific training dataset allows the model to better fit the characteristics of the domain, improving the accuracy and adaptability of sentiment polarity judgment and providing reliable support for deep detection at the model layer.

[0052] Preferably, the semantic similarity model adopts a combination of a bidirectional encoder representation transformation pre-trained model and a graph attention network. The bidirectional encoder representation transformation pre-trained model captures surface features, syntactic features, and semantic features. The graph attention network improves processing performance. A weighting strategy is used to weight different graph attention network layers. The model is trained using a hybrid loss function based on a keyword library.

[0053] Specifically, a combined architecture of a bidirectional encoder representation transformation pre-trained model and a graph attention network is first constructed. The input text is fed into the bidirectional encoder representation transformation pre-trained model, which extracts surface features, syntactic features, and semantic features of the text layer by layer. The extracted features are then input into the graph attention network, and the graph attention mechanism is used to strengthen the weights of key features, thereby improving the model's performance in handling complex semantics. A reasonable weighting strategy is designed to allocate corresponding weights according to the feature extraction capabilities of different graph attention network layers, so that the output features of each layer can be effectively fused. A hybrid loss function based on a keyword database is constructed, which combines keyword matching-related loss terms with traditional loss terms for model training and optimization, thereby improving the accuracy of the model's semantic similarity judgment.

[0054] It should be noted that the architecture of the combined model fully leverages the advantages of the bidirectional encoder representation transformation pre-trained model in feature extraction and the role of the graph attention network in feature enhancement. The application of weight strategies makes feature fusion more reasonable, while the hybrid loss function makes model training more in line with the needs of domain security detection. Ultimately, it significantly improves the feature capture capability and judgment accuracy of the semantic similarity model and enhances the model's adaptability to complex semantic scenarios.

[0055] Preferably, content detection is performed on user requests and domain big model outputs through the rule layer, including: the risk level labels output by the keyword library and prompt word regular expression matching template library in the rule layer include low risk, medium risk, high risk and emergency risk. When the keyword library or prompt word regular expression matching template library outputs medium risk, high risk or emergency risk labels in real time, the answer is rejected or the output is blocked, and the violation content and matching label information are output to the interpretation layer.

[0056] Specifically, during the rule-layer detection process, the system receives user requests and text content output by the domain big data model in real time. It then compares these texts with a keyword matching algorithm and a regular expression matching algorithm, respectively, against a keyword database and a prompt word regular expression matching template database. Based on the comparison results, the rule layer outputs corresponding risk level labels, categorized into low risk, medium risk, high risk, and emergency risk. When the detection result is medium risk, high risk, or emergency risk, a preset interception mechanism is triggered, rejecting the user request and intercepting the output of the big data model. Simultaneously, the system automatically extracts the violating text content and corresponding matching label information, organizes it according to a preset format, and transmits it to the interpretation layer.

[0057] It should be noted that the clear risk level classification makes the detection results more hierarchical, making it easier to quickly determine the severity of the risk; the interception mechanism for medium-high risk and emergency risk can promptly prevent the interactive spread of illegal content and effectively reduce security risks; the synchronization of illegal information and matching tags to the interpretation layer provides an accurate basis for the generation of subsequent interpretation responses, ensuring that the interpretation content is highly consistent with the detection results.

[0058] Preferably, content detection is performed on user requests and domain big model outputs through the model layer, including: the sentiment labels output by the sentiment polarity model in the model layer include normal neutral, positive assistance, negative inducement and malicious attack. When the sentiment polarity label output by the sentiment polarity model in real time is negative inducement or malicious attack, and the semantic similarity judgment result of the semantic similarity model is higher than a set threshold, the answer is refused or the output is blocked, and the illegal content and label information are output to the interpretation layer.

[0059] Specifically, during model-level detection, the text content of the user request and the output of the domain-wide model is first input into the sentiment polarity model. This model analyzes the sentiment tendency of the text and outputs sentiment labels of normal / neutral, positive / supportive, negative / inducing, or malicious attack. Simultaneously, the same text content is input into the semantic similarity model to calculate the semantic similarity between the text and the content in the risk sample library, obtaining a semantic similarity judgment result. The output result of the sentiment polarity model and the judgment result of the semantic similarity model are jointly judged. When the sentiment label is negative / inducing or malicious attack and the semantic similarity judgment result is higher than a set threshold, an interception mechanism is triggered, refusing to answer the user request and intercepting the output of the domain-wide model. Subsequently, information related to the illegal text content, sentiment labels, and semantic similarity judgment is collected, organized, and sent to the interpretation layer.

[0060] It should be noted that the dual judgment strategy of sentiment polarity and semantic similarity effectively avoids the misjudgment problem caused by single-dimensional judgment and improves the accuracy of model layer detection; the interception is triggered only when both conditions are met, which not only ensures the effective interception of malicious content, but also avoids the wrong interception of normal text, ensuring the smoothness of legitimate interaction; and the complete violation-related information is output to the interpretation layer, providing sufficient support for generating accurate and reasonable interpretation responses.

[0061] Accordingly, this embodiment also provides a domain large model content security detection system, including a rule layer, a model layer, and an interpretation layer. The rule layer includes a keyword library and a prompt word regular expression matching template library, with the keywords in the keyword library and the prompt word templates in the prompt word regular expression matching template library labeled with risk levels. The model layer includes a sentiment polarity model and a semantic similarity model, which can set a semantic similarity judgment threshold and can set a combined judgment strategy of sentiment polarity and semantic similarity. The interpretation layer can construct interpretation templates according to the needs of the domain large model scenario. The rule layer, model layer, and interpretation layer are deployed between the user and the domain large model in the form of detectors. The rule layer and model layer perform content detection on user requests and domain large model outputs, and the interpretation layer outputs response information based on the detection results and interpretation templates.

[0062] Specifically, during system construction, functional modules of a rule layer, a model layer, and an interpretation layer are first built: In the rule layer, a keyword library and a prompt word regular expression matching template library are constructed, along with risk level labeling; in the model layer, a sentiment polarity model and a semantic similarity model are integrated, with reserved interfaces for threshold setting and configuration of decision-making strategies; in the interpretation layer, interpretation template libraries corresponding to different detection results are designed based on the application scenarios of the domain-wide model. Then, the three modules are integrated into a detector, deployed on the interactive data link between the user and the domain-wide model, and collaborative detection logic between the rule layer and the model layer is set up. Parallel or serial detection methods can be used to detect user requests and the output of the large model; the interpretation layer receives the detection results from both layers in real time, matches the corresponding template from the interpretation template library, generates standardized and scenario-appropriate response information, and outputs it.

[0063] It should be noted that the layered architecture of this system makes the functions of each module clear and the responsibilities well-defined, facilitating maintenance and upgrades; the deployment method of the detector does not intrude on the core interaction process between the user and the large model, ensuring the smoothness of the interaction; the collaborative detection between the rule layer and the model layer improves the comprehensiveness and accuracy of the detection, and the response output of the interpretation layer allows users to understand the detection results, enhancing the usability and reliability of the system.

[0064] Example 2

[0065] With the rapid development of Large Language Model (LLM) technology, content risk management has become a core issue of common concern in academia and industry, with the core objective of preventing any form of unauthorized information disclosure or misuse. The model's ability to parse input data, the uncontrollability of generated content, and the complexity of information relationships make traditional security detection mechanisms inadequate in dealing with new threats, especially in fields with high security requirements. Content security of large models has become an unavoidable primary issue directly affecting their practical application across various fields.

[0066] Based on this, this embodiment provides a method for content security detection of large domain models. It aims to address the content security needs of large domain models by sorting out the security issues in large domain model scenarios, designing a dynamic update mechanism for sensitive words and a hierarchical coupling method, and establishing a cascaded security detection architecture to achieve content security detection from two dimensions: rules and semantic models.

[0067] Specifically, this method addresses the input content security and output content reliability requirements faced in the practical application of large-scale domain models from two aspects: rules and semantic models. It fully leverages the application scenario information of large-scale domain models and constructs a cascaded content security detection framework of "rule layer + model layer + interpretation layer" from two dimensions: pattern matching and semantic recognition. First, the method identifies the violation scenarios faced by domain models, constructs a keyword library and prompt word regular expression matching templates related to these scenarios, and designs corresponding risk levels based on scenario security requirements. Next, it constructs a "sentiment limit + semantic similarity" model to identify implicit or complex semantics in the content. Finally, it forms a content security detection framework for large-scale domain model scenarios. During detection, the detection conclusions from the rule layer and model layer are output to the interpretation layer to form interpretable detection results.

[0068] Preferably, the domain large model content security detection method of this embodiment includes the following steps:

[0069] Step S1: Identify violation scenarios in the domain's large-scale models, build a keyword library and a prompt word regular expression matching template library, and label the keywords and prompt word templates with risk levels according to scenario requirements;

[0070] Step S2: Conduct offline prompt word attack tests on the large domain model, extract entities or extract regular expression matching templates from the successfully attacked prompt words, and add them to the rule layer sensitive word library or prompt word template library;

[0071] Step S3: Construct a model layer consisting of an emotion polarity model and a semantic similarity model, set a threshold for semantic similarity judgment, and design a strategy to combine emotion polarity and semantic similarity.

[0072] Step S4: Design the discrimination strategy for the rule layer and the model layer, design the hierarchical conclusion coupling strategy for the rule layer and the model layer, and output the conclusion to the interpretation layer;

[0073] Step S5: Construct an explanation template based on the requirements of the domain's large-scale model scenario;

[0074] Step S6: Deploy the constructed detector between the user and the domain big model. The rule layer and the model layer perform content detection on the user request and the big model output according to their respective detection strategies, and output the final detection decision to the interpretation layer according to the hierarchical conclusion coupling strategy. The interpretation layer fills in the information according to the interpretation template and outputs it.

[0075] In step S1, the purpose of identifying violation scenarios in the domain-wide model is to establish the sources of the keyword library and the prompt word regular expression matching template library, ensuring the keyword library and template library are well-suited to the domain scenarios. Preferably, the keyword library construction steps include: multi-source seed word collection, vocabulary enhancement and deduplication, and hierarchical annotation. In this embodiment, the confidential domain is used as an example for illustration.

[0076] 1. Multi-source seed word collection

[0077] The sources of the initial seed word library include three aspects: general specifications, domain regulations and systems, and historical violation data. In terms of general specifications, this embodiment is based on the general regulation framework and refines the boundaries in combination with domain characteristics. Specifically, based on the 5 major categories and 32 sub-categories of security risk dimensions (violating values, infringing rights and interests, commercial violations, etc.) of the corpus and generated content in the "Basic Requirements for the Security of Generative Artificial Intelligence Services" (TC260-003), a keyword library foundation is formed.

[0078] In terms of domain regulations and systems, sort out the confidentiality clauses, and use ERNIE-UIE to extract entities in the confidentiality clauses. ERNIE-UIE can directly extract entities such as the responsible subject and the scope of confidentiality in the "confidentiality clauses" without zero-sample retraining of the model. The Schema configuration can be hierarchically defined as {'classified documents': {'classification level': ['top secret', 'confidential'], 'distribution scope': ['internal organizations']}} to match the nested entity structure in the confidentiality agreement, and add the identified entities to the sensitive words.

[0079] In terms of historical violation data, investigate historical violation case reports, regulatory notifications, user complaints, and historical data of security review reports in the domain, and use the TF-IDF+RAKE combined algorithm to capture high-frequency core vocabulary. It is necessary to first perform text preprocessing and then achieve priority screening through two steps of "candidate word mining - weight ranking". First, use Jieba to split Chinese words, perform word segmentation, stop word removal (removing meaningless words such as "de", "zai", etc.) and词性过滤 (retaining core word types such as nouns and verbs) on the original text to provide clean corpus for the algorithm. Then call the RAKE algorithm, which automatically splits continuous phrases in the text into candidate words instead of single characters by counting word frequencies and analyzing co-occurrence relationships between words. At the same time, based on the logic that the higher the frequency of a word in a phrase and the fewer co-occurrences with other words, the more important it is, a candidate vocabulary pool with both high frequency and independence is initially screened to avoid missing multi-word core expressions. Subsequently, introduce the TF-IDF algorithm to calculate the weights of the candidate words output by RAKE. The TF value reflects the frequency of occurrence of the candidate word in the current text (higher scores for high-frequency words), and the IDF value measures the scarcity of the word in the entire corpus (higher scores for words that are only highly frequent in the target domain and rare in other domains). The product of the two is the final weight. Finally, sort in descending order of weights and preferentially extract the top-weighted words, which not only ensures the high-frequency attribute but also excludes general high-frequency words without domain significance, achieving accurate capture of core vocabulary. Finally, manually verify the core vocabulary selected and add it to the sensitive word library.

[0080] 2. Lexical enhancement and deduplication

[0081] In terms of vocabulary enhancement, this embodiment adopts a technical approach of "large-scale model generation + manual screening" to achieve vocabulary enhancement. First, a structured prompt word is constructed, and a "security and confidentiality expert" role is set, with the task of "generating synonyms, related terms, and scenario-extended words," with additional examples (such as "confidentiality obligor → confidentiality responsible party, agreement signatory"), and the limitation of "only outputting vocabulary suitable for confidentiality clauses" is applied. A large-scale model without security fences is selected, and prompt words and seed words are input, and the generation interface is called to generate candidate words in batches. The model relies on the accumulation of legal corpora to mine professional expressions such as "core algorithm" and "confidential carrier," avoiding the semantic bias of general models. BGE-M3 is used to calculate the vector similarity between candidate words and seed words, retaining words with a threshold ≥0.85, and simultaneously manually checking for misgenerated terms such as "secret → privacy" to ensure domain relevance.

[0082] In terms of deduplication, this embodiment adopts the technical approach of "large model judgment + manual screening" to achieve word deduplication. First, this embodiment organizes the enhanced vocabulary (such as "person responsible for confidentiality" and "person obligated to maintain confidentiality") into standard text without special symbols, unifies the terminology format (such as unifying "declassification period" and "declassification period" into "declassification period"), and removes obviously repetitive literal words (such as deleting the latter from "trade secret" and "trade secret information"). A suitable large-scale model is selected to construct the prompt "As an expert on confidentiality clauses, determine whether two sets of words are semantically repetitive (such as considering 'confidential carrier' and 'confidential carrier' as repetitive), and output a similarity score (0-1)." Preprocessed word pairs are then input in batches. A similarity threshold of ≥0.9 is set (which can be customized according to domain confidentiality requirements). More general expressions among words with scores exceeding the threshold are retained (such as retaining the former from "core algorithm secret" and "algorithm core secret"), and duplicates are deleted. For words whose judgment by the large-scale model is ambiguous (such as "confidentiality period" and "confidentiality duration"), manual confirmation is conducted in conjunction with confidentiality regulations to avoid accidental deletion of professional terms, ultimately forming a highly accurate and non-repetitive keyword library for the confidentiality domain.

[0083] 3. Hierarchical labeling

[0084] This embodiment employs a "large-scale model generation + manual screening" technical approach to achieve risk-level labeling of the keyword database. First, referencing the "Secrecy Law" and standard practices in the field of secrecy, a four-level standard (Top Secret, Confidential, Secret, Internal) is established, classifying risk levels into low, medium, high, and urgent. A suitable large-scale model is then used to construct prompts: "As a secrecy review expert, label keywords according to 'low / medium / high / urgent,' explaining the basis (e.g., 'core technical parameters' → high)." The keyword database is then input into the model in batches, and the model outputs the risk level for each keyword. Manual sampling is then used to verify whether the labeling accuracy meets the preset standards.

[0085] Preferably, the construction of the prompt word regular expression matching template library in this embodiment includes: risk type definition, malicious prompt sample collection, feature extraction and transformation, and hierarchical labeling, as detailed below.

[0086] 1. Definition and classification of risk types

[0087] Based on the "attack purpose," risks in the confidential domain are divided into four categories: induced disclosure (attempting to obtain classified information), circumventing audits (bypassing security mechanisms to output classified content), blurring boundaries (confusing classified and non-classified information), and model theft (attempting to obtain model information), and corresponding to information levels. In combination with the needs of the domain scenario, the risk levels are divided into four levels: mostly harmless, potentially harmful, harmful, and extremely harmful.

[0088] 2. Collection of Malicious Prompt Samples

[0089] This embodiment employs a "large-scale model generation + manual screening" approach to collect malicious alert samples. First, a large-scale model without deployed security fences is selected, and targeted alert words are designed to "simulate malicious users, combine four types of risks in the confidentiality domain, and generate attack scripts, each corresponding to a specific confidentiality scenario," thus initially obtaining basic samples. Second, historical large-scale model interaction logs are reviewed, real-world cases of past attacks on the confidentiality large-scale model are collected, and scripts adapted to confidentiality scenarios are injected into publicly available alert words in attack cases. Real scripts adapted to these scenarios are extracted, and valid cases matching the domain scenario are selected. Finally, all samples are deduplicated and categorized into the four risk scenarios.

[0090] 3. Feature Extraction and Transformation

[0091] First, the labeled malicious alert samples are categorized by risk type (induced disclosure, circumvention of review, blurred boundaries, model theft). Keywords are extracted using jieba, and sentence structures are annotated. TF-IDF is used to extract high-frequency words for each category, forming a list of linguistic features for each risk type. For example, the induced disclosure category includes "obtain / provide / inform" + classified terms, and the circumvention of review category includes "in the name of .* / bypass .* review." Second, regular expressions are used to convert the feature list. For example, the circumvention of review feature "in the name of .*.* classified" is transformed into "in the name of [^,。]{1,10}[^,。]{1,15} classified," with character length limited to avoid generic matching. The induced disclosure feature "provide .* core secrets" is transformed into "provide .* core (secret|confidential|data)." These are then organized into an initial template library according to risk type.

[0092] 4. Hierarchical labeling

[0093] The method for tagging prompt word templates in this embodiment is the same as the technical approach for tagging keyword databases, both employing a "large model generation + manual screening" approach to achieve hierarchical tagging. Templates are tagged according to four risk levels: low, medium, high, and urgent. Details are omitted here. Manual sampling is used to verify whether the recall rate and false positive rate meet preset standards.

[0094] Preferably, in step S2, attack prompt words and variations of historical attack case prompt words in the public library are continuously collected, offline attack tests are carried out on the domain large model, and high-frequency core words are captured from the prompt words of successful attacks using the TF-IDF+RAKE combined algorithm. After the core words are marked with risk level, they are added to the keyword library. At the same time, prompt word regular expression matching templates are extracted, and after marking the risk scenarios and risk levels, they are added to the prompt word template library.

[0095] In step S3, considering that a single semantic analysis model may not be able to fully uncover hidden and complex semantics, and that sentiment polarity also contributes to semantic determination, this embodiment uses a two-level model of "sentiment polarity model + semantic similarity model" to achieve semantic discrimination at the model layer.

[0096] Preferably, regarding the sentiment polarity model, considering the balance between training and inference costs and accuracy, this embodiment adopts a technical approach based on fine-tuning the DistilBERT model using a self-built dataset. First, a sentiment polarity dataset containing 2000 samples was constructed. Based on existing research, four labels were used: "normal neutral," "positive assistance," "negative inducement," and "malicious attack." Each sample was labeled with an "instruction word-sentiment association pair," creating a dataset adapted to the characteristics of the prompt words. The training and test sets were divided in a 7:3 ratio. The Chinese pre-trained version, distilbert-base-chinese, was selected and loaded into the Hugging FaceTransformers framework. The default classification head was modified to a 4-class output layer (corresponding to the four sentiment polarities). A new "instruction word embedding layer" was added to convert the labeled instruction words into vectors and integrate them into the model, enhancing the semantic capture of key instructions such as "ignore" and "provide." The cross-entropy loss function was used, with AdamW as the optimizer and a learning rate of 2e-5. During training, a "sentiment keyword masking strategy" was incorporated, randomly masking negative words such as "fraud" and "steal" and positive words such as "assist" and "optimize," forcing the model to learn sentiment association features.

[0097] Preferably, regarding the semantic similarity model, this embodiment employs a semantic similarity recognition model based on BERT and a Graph Attention Network (GAT). This model utilizes information captured by BERT at different levels, including surface features, syntactic features, and semantic features, and then leverages the GAT network to improve algorithm performance. The hidden state matrices and attention matrices of multiple intermediate layers of BERT are used as the node feature matrices and adjacency matrices for the corresponding number of GAT layers, respectively. A weighting strategy is employed to weight different GAT layers, and a hybrid loss function based on a keyword database is designed to drive the model to pay more attention to domain-infringing keywords during training.

[0098] Preferably, the decision strategies for the rule layer and model layer designed in step S4 are as follows:

[0099] Rule-based judgment strategy: The rule layer contains two rule bases: a rule database of illegal keywords and a rule database of malicious prompt word templates, which correspond to four risk levels: low, medium, high, and urgent. When either rule base outputs a "medium / high / urgent" label, regardless of the risk label output by the other rule base, the answer will be rejected / output will be blocked, and the illegal content and the matched label information will be output to the interpretation layer.

[0100] Model-level decision-making strategy: The model layer includes the outputs of two models: sentiment polarity determination and semantic similarity determination. The sentiment polarity model output corresponds to four categories of labels: "normal / neutral," "positive / supportive," "negative / inducing," and "malicious attack." Semantic similarity determination is based on a threshold, which is set according to the security requirements of the domain scenario and includes both cases above and below the threshold. When the sentiment polarity is "negative / inducing" and the similarity is higher than the threshold, a refusal to answer / output interception is executed, and the violation content and label information are output to the interpretation layer.

[0101] Preferably, the data flow of this method is illustrated as follows: Figure 2 As shown, the data first enters the keyword library of the rule layer for matching. If a match is successful, the answer is rejected and the relevant information is output to the interpretation layer, which selects the corresponding template for interpretation. If the match is unsuccessful, the data enters the prompt word template library for matching. If the prompt word template library matches successfully, the answer is rejected and the relevant information is output to the interpretation layer, which selects the corresponding template for interpretation. If the match is unsuccessful, the data enters the model layer. Then, the data is judged by the sentiment polarity model and the semantic similarity model respectively. If the model layer's discrimination strategy is satisfied, the answer is rejected and the relevant information is output to the interpretation layer, which selects the corresponding template for interpretation. If the model layer's discrimination strategy is not satisfied, the main model responds normally. It is worth noting that, to prevent unexpected situations, this method sets up a bypass, which can bypass the above detection path.

[0102] In summary, this method performs content detection and interception on large-scale domain models from a rule layer, a model layer, and an interpretation layer. By constructing a rule layer filter, it identifies potential violation scenarios that large-scale domain models may face, establishes a sensitive word library and regular expression matching templates, and implements a dynamic update mechanism based on domain scenarios to combat malicious exploitation, potential abuse, and prompt word attacks against large-scale domain models. It also classifies corresponding risk levels based on business needs. By constructing a model layer containing a semantic model, it performs semantic recognition and processing of content. Finally, by constructing interpretation templates, it fills in relevant information based on the outputs of the rule layer and the model layer. This ultimately forms a three-layer structure for large-scale model content security detection, analyzing and judging whether the large-scale model meets the security requirements of output content in sensitive prompt word scenarios, thereby improving the security and reliability of large-scale domain models.

[0103] Example 3

[0104] This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the domain large model content security detection method of Embodiment 1 or 2. The computer program can be in the form of source code, object code, executable file, or some intermediate form.

[0105] Example 4

[0106] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the domain-wide model content security detection method of Embodiment 1 or 2. The computer program can be in the form of source code, object code, executable file, or some intermediate form. The storage medium includes any entity or device capable of carrying computer program code, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0107] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

[0108] It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

Claims

1. A method for content security detection in a large domain model, characterized in that, include: A rule layer is constructed, which includes a keyword library and a prompt word regular expression matching template library. Risk levels are marked for the keywords in the keyword library and the prompt word templates in the prompt word regular expression matching template library. A model layer is constructed, which includes an emotional polarity model and a semantic similarity model. A semantic similarity judgment threshold is set, and a judgment strategy combining emotional polarity and semantic similarity is set. Construct an interpretation layer and create interpretation templates based on the needs of the domain's large-scale model scenario; The rule layer, model layer, and interpretation layer are constructed as detectors and deployed between the user and the domain big model; the rule layer and model layer perform content detection on user requests and domain big model outputs, and the interpretation layer outputs response information based on the detection results and interpretation templates; The sentiment polarity model is implemented using a lightweight bidirectional encoder representation transformation pre-trained model fine-tuned based on the training dataset; The semantic similarity model adopts a combination of a bidirectional encoder representation transformation pre-trained model and a graph attention network. The bidirectional encoder representation transformation pre-trained model captures surface features, syntactic features and semantic features. The graph attention network improves processing performance. A weight strategy is used to weight different graph attention network layers. The model is trained using a hybrid loss function based on a keyword library. Content detection is performed on user requests and domain big model outputs through the rule layer, including: the risk level labels output by the keyword library and prompt word regular expression matching template library in the rule layer include low risk, medium risk, high risk and emergency risk. When the keyword library or prompt word regular expression matching template library outputs medium risk, high risk or emergency risk labels in real time, the answer is rejected or the output is blocked, and the violation content and matching label information are output to the interpretation layer. Content detection is performed on user requests and domain-wide model outputs through the model layer. This includes: the sentiment polarity model outputs sentiment labels including normal neutral, positive support, negative inducement, and malicious attack. When the sentiment polarity model outputs a negative inducement or malicious attack in real time, and the semantic similarity judgment result of the semantic similarity model is higher than a set threshold, the system will refuse to answer or block the output, and output the violation content and label information to the interpretation layer.

2. The method for content security detection of a large domain model according to claim 1, characterized in that, The rule construction layer includes: constructing a keyword library through multi-source seed word collection, vocabulary enhancement and deduplication, and constructing a prompt word regular expression matching template library through risk type definition, malicious prompt sample collection, and feature extraction and transformation.

3. The method for content security detection of a large domain model according to claim 1, characterized in that, The rule construction layer also includes offline prompt word attack testing: collecting attack prompt words and their entity variants, conducting offline attack tests on a large domain model, extracting core words and regular expression matching templates from successfully attacked prompt words, marking risk levels and risk scenarios respectively, and adding them to the keyword library and prompt word regular expression matching template library accordingly.

4. A domain-wide large-scale model content security detection system, characterized in that, It includes a rule layer, a model layer, and an interpretation layer; The rule layer includes a keyword library and a prompt word regular expression matching template library. The keywords in the keyword library and the prompt word templates in the prompt word regular expression matching template library are marked with risk levels. The model layer includes an emotional polarity model and a semantic similarity model, and can set a semantic similarity judgment threshold and a combined judgment strategy of emotional polarity and semantic similarity. The interpretation layer can construct interpretation templates according to the needs of a large domain model scenario. The rule layer, model layer, and interpretation layer are deployed between the user and the domain big model in the form of detectors. The rule layer and model layer perform content detection on user requests and the output of the domain big model, and the interpretation layer outputs response information based on the detection results and interpretation templates. The sentiment polarity model is implemented using a lightweight bidirectional encoder representation transformation pre-trained model fine-tuned based on the training dataset; The semantic similarity model adopts a combination of a bidirectional encoder representation transformation pre-trained model and a graph attention network. The bidirectional encoder representation transformation pre-trained model captures surface features, syntactic features and semantic features. The graph attention network improves processing performance. A weight strategy is used to weight different graph attention network layers. The model is trained using a hybrid loss function based on a keyword library. The rule layer performs content detection on user requests and the output of the domain big model, including: the risk level labels output by the keyword library and prompt word regular expression matching template library in the rule layer include low risk, medium risk, high risk and emergency risk. When the keyword library or prompt word regular expression matching template library outputs medium risk, high risk or emergency risk labels in real time, the answer is rejected or the output is blocked, and the violation content and matching label information are output to the interpretation layer. The model layer performs content detection on user requests and the output of the domain big model, including: the sentiment labels output by the sentiment polarity model in the model layer include normal neutral, positive assistance, negative inducement and malicious attack. When the sentiment polarity label output by the sentiment polarity model in real time is negative inducement or malicious attack, and the semantic similarity judgment result of the semantic similarity model is higher than the set threshold, the answer is refused or the output is blocked, and the violation content and label information are output to the interpretation layer.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the domain large model content security detection method according to any one of claims 1-3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the domain large model content security detection method according to any one of claims 1-3.

Images