A large model violation semantic detection method, system, device and medium

Abstract

This invention relates to the field of large-scale model security technology, and discloses a method, system, device, and medium for detecting illegal semantics in large-scale models. The method includes: acquiring multiple levels of textual semantic features through a bidirectional encoder representation transformation model, and acquiring textual temporal features through a gated recurrent unit; constructing an illegal word library and setting a security risk attention mechanism based on the illegal word library to obtain a security risk feature matrix; concatenating the hidden state matrix of the model's target layer, the textual temporal features, and the security risk feature matrix to obtain a combined feature matrix, and weighting the attention matrix of the model's target layer, inputting both into the corresponding graph attention network to obtain a fused output; training the model using a violation hybrid loss function based on the illegal word library, and then performing large-scale model illegal semantic detection. This invention, through multi-dimensional technical design and model architecture optimization, achieves significant improvements in the accuracy, comprehensiveness, and adaptability of illegal semantic detection.

Description

Technical Field

[0001] This invention relates to the field of large model security technology, and in particular to a method, system, device and medium for detecting illegal semantics in large models. Background Technology

[0002] With the continuous development of internet technology, the scale of user-generated content (UGC) is constantly expanding, and its application scenarios are becoming increasingly diverse, widely covering multiple fields such as social interaction, information sharing, and content creation. This type of content is not only rich and varied in form, but also often accompanied by subtle expressions and deliberate evasion techniques, posing significant challenges to content security detection.

[0003] Traditional content security detection technologies often rely on fixed matching rules or shallow learning models, with their core logic essentially being the identification of specific patterns. While these methods can handle clearly defined violations, they easily become ineffective against flexible circumvention methods such as word substitution, semantic transformation, and splitting and recombining. They struggle to accurately capture the true intent behind the content, leading to frequent missed detections. Furthermore, due to a lack of deep understanding and integration of contextual semantics, they cannot accurately distinguish the differences in meaning between words in different contexts, often misjudging normal communication and discussion as violations. This not only impacts user experience but also increases the cost of subsequent review.

[0004] Today, the presentation of illegal content is becoming increasingly covert, contextualized, and semantically charged. It is no longer limited to expressions containing explicitly sensitive elements, but increasingly conveys illegal semantics through metaphors, scene associations, and fictional stories. This type of content appears to conform to normal expression logic, but it hides potential security risks. Traditional detection methods, due to their inability to overcome the gap in semantic cognition, are unable to effectively identify this new type of illegal content and are no longer sufficient to meet the actual needs of current content security detection.

[0005] To address the aforementioned issues, some detection technologies based on large models have been attempted in related fields. For example, patent document CN119322978A discloses a method for detecting illegal content using a large model. This method includes: S1: inputting the user's content generation request into a Transformer model, where each layer of the Transformer model outputs a latent feature sequence, wherein the input of the current layer is the latent feature sequence output by the previous layer; S2: using an optimized spatiotemporal sequence discrimination model to analyze the violation probability of the latent feature sequences output by each layer; S3: inputting the violation probability of the latent feature sequences output by each layer into a trained secondary learner, and outputting the final violation probability.

[0006] The aforementioned large-scale model-based content violation detection method extracts the latent feature sequences output from each layer of the model, analyzes the violation probability using a discriminative model, and then outputs the final result through a secondary learner. However, this method fails to fully integrate the temporal correlation features and core risk information of the text itself, does not delve deeply enough into the semantic deep correlation, and does not adequately consider the differentiated value of features at different levels in detection. As a result, the detection accuracy needs to be improved when dealing with obscure or mutated content violations.

[0007] For example, patent document CN121211228A discloses a public opinion information detection method based on a heterogeneous large model. This method acquires public opinion text and performs preprocessing such as sentence segmentation, denoising, and word segmentation. The processed text is then input into a detection model to output harmful information categories and warning levels. The detection model consists of a first large language model and a second large language model, possessing structural features such as cross-architecture semantic alignment, hierarchical knowledge distillation, domain-enhanced attention, and multi-channel decision fusion. Specifically, cross-architecture semantic alignment achieves hidden space sharing through bidirectional projection; hierarchical knowledge distillation dynamically allocates weights based on the contribution of each layer's tasks; domain-enhanced attention introduces domain bias to strengthen semantic focus; and multi-channel decision fusion adaptively integrates the outputs of the two models and classifies them.

[0008] The aforementioned public opinion information detection method uses a dual-model architecture combined with multiple fusion mechanisms for detection. However, the model structure of this method is relatively complex, lacks flexibility in adapting to different application scenarios, and lacks a strengthening mechanism for the core features of violations. When faced with diverse expressions of violations, it is prone to identification bias. Furthermore, the decision fusion process fails to fully consider the specificity of violation risks and fails to achieve efficient and accurate identification of various types of violation semantics. The overall detection effect still has room for improvement. Summary of the Invention

[0009] To address the issues of semantic feature information effectiveness and security risk attention in existing technologies, this invention proposes a large-scale model violation semantic detection method, system, device, and medium. Addressing content security needs, it constructs a multi-layered framework for fusing text semantic features and text temporal features based on the importance of different levels of text semantic features, and designs a security risk attention mechanism based on a violation lexicon, achieving large-scale model violation semantic detection from both word and sentence dimensions.

[0010] The technical solution adopted in this invention is as follows:

[0011] A method for detecting semantic violations in large models, comprising:

[0012] Multiple levels of text semantic features are obtained through a bidirectional encoder representation transformation model, and text temporal features are obtained through a gated recurrent unit; a violation word library is constructed, and a security risk attention mechanism based on the violation word library is set to obtain a security risk feature matrix;

[0013] The hidden state matrix of the target layer of the bidirectional encoder representation transformation model, the text temporal features, and the security risk feature matrix are concatenated to obtain a combined feature matrix; the attention matrix of the target layer of the bidirectional encoder representation transformation model is weighted according to the importance of text hierarchical information to obtain a weighted attention matrix.

[0014] The combined feature matrix and the weighted attention matrix are input into the corresponding graph attention network. The outputs of each layer of the graph attention network are fused according to the attention-normalized weights to obtain the fused output.

[0015] A violation semantic detection model is trained based on the violation lexicon and a violation hybrid loss function. The violation semantic detection model includes a bidirectional encoder representation transformation model, a gated recurrent unit, and a graph attention network. The trained violation semantic detection model is then used to perform large-scale violation semantic detection.

[0016] Furthermore, before obtaining the semantic features of the text, the sentences in the training phase are segmented, including: adding a tag at the beginning of the sentence to aggregate sentence-level features, adding a tag at the end of the sentence to distinguish sentence boundaries, mapping each segmented word to a word embedding, and superimposing position embedding and segment embedding to form an initial input vector matrix.

[0017] Furthermore, the initial sources of the violation term database include general specifications, domain regulations and systems, and historical violation data. The historical violation data includes publicly available historical case data, historical violation data within application scenarios, and expanded historical violation data. The method for generating the expanded historical violation data includes: supplementing the core terms with derivative terms, variant terms, and vivid expressions through in-depth mining of historical violation data based on the compliance framework.

[0018] Furthermore, the security risk attention mechanism based on the violation word library includes: querying whether the input character is a violation word in the violation word library, and assigning a corresponding feature vector to each input character; calculating a security risk attention score based on the input character and its corresponding feature vector, normalizing the security risk attention score to obtain a normalized security risk attention distribution; calculating a security risk vector based on the normalized security risk attention distribution and the input character, and converting the security risk vector into a security risk feature matrix.

[0019] Furthermore, the step of weighting the attention matrix of the target layer of the bidirectional encoder representation transformation model according to the importance of text-level information includes: setting the weight coefficients of the attention matrix of each target layer according to the logic of lexical information importance < syntactic information importance < semantic information importance, with the sum of all weight coefficients being 1, and the value range of each weight coefficient being 0 to 1.

[0020] Furthermore, the step of inputting the combined feature matrix and the weighted attention matrix into the corresponding graph attention network includes: the number of layers in the graph attention network is consistent with the number of target layers in the bidirectional encoder representation transformation model, and each layer of the graph attention network corresponds to a set of combined feature matrices and weighted attention matrices of the target layers.

[0021] Furthermore, the violation hybrid loss function includes a bidirectional encoder representation transformation model loss function, a graph attention network loss function, and a violation binary loss function; the violation binary loss function is constructed based on the binary cross-entropy function and is used to learn the positional distribution information of violation words in the input text, and the input contains a binary attention vector of the same length as the text sequence, and the binary attention vector is labeled according to the violation word library to indicate whether the input characters are violation words.

[0022] A large-scale model violation semantic detection system, comprising:

[0023] The feature extraction module is configured to obtain multiple levels of text semantic features through a bidirectional encoder representation transformation model and obtain text temporal features through a gated recurrent unit; construct a violation word library and set a security risk attention mechanism based on the violation word library to obtain a security risk feature matrix;

[0024] The feature concatenation module is configured to concatenate the hidden state matrix of the target layer of the bidirectional encoder representation transformation model, the text temporal features, and the security risk feature matrix to obtain a combined feature matrix; and to weight the attention matrix of the target layer of the bidirectional encoder representation transformation model according to the importance of text hierarchical information to obtain a weighted attention matrix.

[0025] The feature fusion module is configured to input the combined feature matrix and the weighted attention matrix into the corresponding graph attention network, and fuse the outputs of each layer of the graph attention network according to the attention-normalized weights to obtain the fused output;

[0026] The model training and application module is configured to train a violation semantic detection model using a violation hybrid loss function based on the violation lexicon, the violation semantic detection model including a bidirectional encoder representation transformation model, a gated recurrent unit, and a graph attention network; and to perform large-scale violation semantic detection using the trained violation semantic detection model.

[0027] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the large model violation semantic detection method.

[0028] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the large model violation semantic detection method.

[0029] The beneficial effects of this invention are as follows:

[0030] This invention significantly improves the accuracy, comprehensiveness, and adaptability of illegal semantic detection through multi-dimensional technical design and model architecture optimization, while effectively making up for the shortcomings of existing related technologies, as detailed below.

[0031] 1. This invention fully explores the multi-level value and temporal correlation characteristics of text semantics. It obtains semantic features at different levels through a bidirectional encoder representation transformation model and captures the temporal evolution information of the text by combining a gated recurrent unit. This allows the model to understand the content from two dimensions: static semantic level and dynamic contextual correlation. Compared with existing technologies that rely on a single feature sequence or lack temporal feature integration, it can more comprehensively capture the deep semantic logic behind the text. It significantly improves the sensitivity to implicit expressions and scene-related violations, effectively bridging the semantic cognition gap faced by traditional methods and some existing technologies.

[0032] 2. This invention constructs a violation word database and designs a corresponding security risk attention mechanism and violation hybrid loss function, strengthening the core features of violations throughout the entire process of feature extraction and model training. This design not only allows the model to actively focus on high-risk information during the learning process, but also accurately identifies variant and hidden forms of violation-related expressions. Compared with existing technologies that lack targeted risk enhancement mechanisms, it significantly reduces the problem of false negatives caused by violation word avoidance methods, while improving the specificity of violation semantic recognition, reducing the misjudgment of normal content, and balancing the rigor and flexibility of detection.

[0033] 3. This invention innovatively employs a processing method combining a combined feature matrix and a weighted attention matrix with a graph attention network. By assigning differentiated weights to features at different levels and performing normalized fusion, it fully respects the differences in the contribution of various features in the detection task, avoiding the problems of some techniques lacking focus in feature utilization or having a coarse fusion mechanism. This refined feature processing and fusion method allows the model to accurately screen key risk information and filter redundant interference, further improving the accuracy and efficiency of illegal semantic detection.

[0034] 4. This invention balances the comprehensiveness of feature extraction with the targeted nature of mechanism design. It can achieve efficient detection without complex heterogeneous architecture. Compared with existing technologies that are cumbersome in structure and lack scene adaptability, the model of this invention maintains detection performance while having stronger scene adaptability. It can cope with diverse user-generated content scenarios, and the training and inference processes are more efficient, which is more in line with the needs of practical applications. It provides more reliable technical support for solving the current problems of concealed, contextualized, and semantic illegal content detection. Attached Figure Description

[0035] Figure 1 This is a flowchart of a large-scale model violation semantic detection method according to Embodiment 1 of the present invention. Detailed Implementation

[0036] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments are now described. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0037] Example 1

[0038] like Figure 1 As shown, this embodiment provides a method for detecting semantic violations in large models, including:

[0039] Multiple levels of text semantic features are obtained through a bidirectional encoder representation transformation model, and text temporal features are obtained through a gated recurrent unit; a violation word library is constructed, and a security risk attention mechanism based on the violation word library is set up to obtain a security risk feature matrix;

[0040] The hidden state matrix, text temporal feature matrix, and security risk feature matrix of the target layer of the bidirectional encoder representation transformation model are concatenated to obtain a combined feature matrix; the attention matrix of the target layer of the bidirectional encoder representation transformation model is weighted according to the importance of text hierarchical information to obtain a weighted attention matrix.

[0041] The combined feature matrix and the weighted attention matrix are input into the corresponding graph attention network. The outputs of each layer of the graph attention network are fused according to the attention-normalized weights to obtain the fused output.

[0042] A violation semantic detection model is trained using a violation hybrid loss function based on a violation lexicon. The violation semantic detection model includes a bidirectional encoder representation transformation model, a gated recurrent unit, and a graph attention network. The trained violation semantic detection model is then used to perform large-scale violation semantic detection.

[0043] It should be noted that this method uses multiple models to work together to comprehensively extract semantic, temporal, and security risk features of the text. At the same time, through weighted fusion and graph attention network processing, it can comprehensively capture violation-related information in the text, improve the model's ability to identify violation semantics, and ensure the accuracy and comprehensiveness of the detection results.

[0044] Preferably, before obtaining the semantic features of the text, the sentences in the training phase are segmented, including: adding a marker at the beginning of the sentence to aggregate sentence-level features, adding a marker at the end of the sentence to distinguish sentence boundaries, mapping each segmented word to a word embedding, and superimposing position embedding and segment embedding to form an initial input vector matrix.

[0045] Specifically, during the model training phase, the input sentence is first segmented into multiple independent segmentation units. Then, a specific marker is added to the beginning of the segmented sentence; this marker is used for subsequent aggregation and extraction of sentence-level overall features by the model. Another specific marker is added to the end of the sentence to clearly distinguish the boundaries between different sentences and avoid information confusion between sentences. Next, each segment is converted into a corresponding word embedding vector, which represents the semantic information of the segment. Then, a positional embedding vector is added to each segment to reflect its position in the sentence, and a segmental embedding vector is added to distinguish different text paragraphs. Finally, the word embeddings, positional embeddings, and segmental embeddings are superimposed and combined to form the initial input vector matrix required by the model.

[0046] It should be noted that adding specific markers helps the model better capture the overall features of the sentence and distinguish sentence boundaries. The combination of word embedding, position embedding and segment embedding can enrich the information dimension of the initial input, lay the foundation for the accurate extraction of subsequent semantic features, and improve the model's ability to understand the semantics of the text.

[0047] Preferably, the initial sources of the violation term database include general specifications, domain regulations and systems, and historical violation data. The historical violation data includes publicly available historical case data, historical violation data within application scenarios, and expanded historical violation data. The method for generating expanded historical violation data includes: supplementing the core terms with derivative terms, variant terms, and vivid expressions through in-depth mining of historical violation data based on the compliance framework.

[0048] Specifically, the process begins with a broad collection of relevant terms related to the definition of violations from general standards. This involves reviewing corresponding regulations and systems across various fields, extracting explicitly prohibited expressions related to violations, and collecting historical case data published through public channels and past violation data in specific application scenarios. This data forms the initial foundation for the violation terminology database. For expanded historical violation data, guided by the established compliance framework, in-depth analysis and mining are conducted on the collected historical violation data to identify core violation terms. This is followed by expanding upon these core terms with related derivatives, variants, and vivid expressions appearing in the current context. These expanded terms are then added to the violation terminology database to form a complete database.

[0049] It should be noted that the initial sources from multiple channels ensure the comprehensiveness and authority of the violation term database, while the supplementary data on historical violations enables the database to promptly cover newly emerging violation expressions, enhancing its timeliness and completeness, and providing sufficient and accurate vocabulary support for the effective operation of subsequent security risk attention mechanisms and violation semantic detection.

[0050] Preferably, a security risk attention mechanism based on a violation word library is set up, including: querying whether the input character is a violation word in the violation word library, and assigning a corresponding feature vector to each input character; calculating a security risk attention score based on the input character and its corresponding feature vector, normalizing the security risk attention score to obtain a normalized security risk attention distribution; calculating a security risk vector based on the normalized security risk attention distribution and the input character, and converting the security risk vector into a security risk feature matrix.

[0051] Specifically, in setting up the security risk attention mechanism, firstly, each character of the input text is compared with words in the violation word database to determine whether the character belongs to a violation word. Based on the query results, a corresponding feature vector is assigned to each input character. The feature vectors of violation characters differ from those of non-violation characters to reflect their security risk attributes. Next, combining the information of the input character itself and its corresponding feature vector, a security risk attention score is obtained for each character through a preset calculation method. This score measures the degree of association between the character and the violation semantics. Then, the security risk attention scores of all characters are normalized to ensure they fall within a uniform range, resulting in a normalized security risk attention distribution. Finally, based on this normalized distribution and relevant information of the input characters, a security risk vector corresponding to each character is calculated. These security risk vectors of all characters are then arranged and combined in sequence to transform into a security risk feature matrix.

[0052] It should be noted that by query matching and feature vector allocation, potential illegal characters in the input text can be accurately located. The calculation and normalization of security risk attention scores can quantify the degree of illegal association of characters. The resulting security risk feature matrix can clearly characterize the distribution of security risks in the text, providing key security risk information for the subsequent construction of combined feature matrices.

[0053] Preferably, the attention matrix of the target layer of the bidirectional encoder representation transformation model is weighted according to the importance of text-level information, including: setting the weight coefficients of the attention matrix of each target layer according to the logic of lexical information importance < syntactic information importance < semantic information importance, the sum of all weight coefficients is 1, and the value range of each weight coefficient is 0 to 1.

[0054] Specifically, firstly, it is clarified that each target layer of the bidirectional encoder representation transformation model corresponds to the lexical, syntactic, and semantic information of the text, respectively. Following the logical relationship that lexical information is less important than syntactic information, and syntactic information is less important than semantic information, corresponding weight coefficients are assigned to the attention matrices of each target layer. This ensures that the sum of the weight coefficients of all target layers is 1, and that the value of each weight coefficient is between 0 and 1. Then, the attention matrix of each target layer is multiplied by its corresponding weight coefficient, and the results are summed to obtain the weighted attention matrix.

[0055] It should be noted that setting weight coefficients based on the differences in the importance of text-level information allows the model to focus more on key semantic information during processing, while also taking into account lexical and syntactic information. This enables the weighted attention matrix to highlight the importance of core information, improve the model's ability to capture deep semantics of the text, and thus optimize the performance of violation semantic detection.

[0056] Preferably, the combined feature matrix and the weighted attention matrix are input into the corresponding graph attention network, including: the number of layers in the graph attention network is consistent with the number of target layers in the bidirectional encoder representation transformation model, and each layer of the graph attention network corresponds to a set of combined feature matrices and weighted attention matrices of the target layer.

[0057] Specifically, firstly, the number of layers in the graph attention network is determined based on the number of target layers in the bidirectional encoder representation transformation model, ensuring that the number of layers remains consistent. Then, a corresponding set of combined feature matrices and weighted attention matrices are assigned to each layer of the graph attention network; that is, the combined feature matrix and weighted attention matrix corresponding to each target layer are input one-to-one into each layer of the graph attention network. Each layer of the graph attention network processes the input feature matrix and attention matrix independently, using the attention mechanism to mine the correlation information between features, strengthening relevant features and suppressing irrelevant features.

[0058] It should be noted that the corresponding setting of the number of layers in the graph attention network and the number of target layers in the bidirectional encoder representation transformation model enables hierarchical processing of features, allowing each network layer to focus on information mining of the corresponding target layer, making full use of the feature advantages of each target layer, improving the targeting and effectiveness of feature processing, and providing high-quality intermediate results for subsequent feature fusion.

[0059] Preferably, the violation hybrid loss function includes a bidirectional encoder representation transformation model loss function, a graph attention network loss function, and a violation binary loss function. The violation binary loss function is constructed based on the binary cross-entropy function and is used to learn the positional distribution information of violation words in the input text. The input contains a binary attention vector of the same length as the text sequence. The binary attention vector is used to label whether the input characters are violation words according to the violation word library.

[0060] Specifically, when constructing the violation hybrid loss function, the bidirectional encoder representation transformation model loss function, the graph attention network loss function, and the violation binary loss function are linearly weighted and combined to form a comprehensive loss function. The violation binary loss function is designed based on the binary cross-entropy function, and its core purpose is to allow the model to learn the positional distribution of violation words in the input text. The input of this loss function includes a binary attention vector of the same length as the input text sequence. This vector is obtained by querying a violation word database and labeling each input character. The labeling results are in binary form, corresponding to whether a character is a violation word. During model training, the error between the model's prediction results and the true labels is calculated using the violation hybrid loss function, and the model parameters are adjusted through backpropagation.

[0061] It should be noted that the violation hybrid loss function integrates the loss information of multiple model components, which can comprehensively reflect the training status of each part of the model and ensure the collaborative optimization of each component; while the introduction of the violation binary loss function enables the model to focus on the positional distribution of violation words, enhance the model's sensitivity to violation words, improve the model's ability to locate and identify violation semantics, and accelerate the training convergence speed of the model.

[0062] Accordingly, this embodiment also provides a large-model semantic violation detection system, including:

[0063] The feature extraction module is configured to obtain multiple levels of text semantic features through a bidirectional encoder representation transformation model and obtain text temporal features through a gated recurrent unit; construct a violation word library and set a security risk attention mechanism based on the violation word library to obtain a security risk feature matrix;

[0064] The feature concatenation module is configured to concatenate the hidden state matrix, text temporal features, and security risk feature matrix of the target layer of the bidirectional encoder representation transformation model to obtain a combined feature matrix; and to weight the attention matrix of the target layer of the bidirectional encoder representation transformation model according to the importance of text hierarchical information to obtain a weighted attention matrix.

[0065] The feature fusion module is configured to input the combined feature matrix and the weighted attention matrix into the corresponding graph attention network, and fuse the outputs of each layer of the graph attention network according to the attention-normalized weights to obtain the fused output;

[0066] The model training and application module is configured to train a violation semantic detection model using a violation hybrid loss function based on a violation lexicon. The violation semantic detection model includes a bidirectional encoder representation transformation model, a gated recurrent unit, and a graph attention network. The module then performs large-scale violation semantic detection using the trained violation semantic detection model.

[0067] It should be noted that the system, through its modular design, clearly divides the various processes of illegal semantic detection, and the modules work together to ensure the orderliness and efficiency of the entire detection process. At the same time, the functional design of each module closely corresponds to the aforementioned detection methods, which can give full play to the technical advantages of the methods and achieve accurate and efficient detection of illegal semantics in large models.

[0068] Example 2

[0069] This embodiment provides a large-scale violation semantic detection method, designing a security risk attention-oriented large-scale violation semantic detection model. This model is based on Bidirectional Encoder Representation Transform (BERT), Graph Attention Network (GAT), and Gated Recurrent Unit (GRU) to identify violation content in text from both word and sentence dimensions. The method first uses BERT to capture different levels of textual semantic features of key interest. Then, it uses GRU to extract temporal features of the text and concatenates these temporal features with the textual semantic features. Next, it integrates different levels of textual information into the corresponding GAT, constructs a customizable violation lexicon, and designs a violation-word-driven security risk attention and violation hybrid loss function to drive the model to pay more attention to violation words during training, ultimately completing the construction of the large-scale violation semantic detection model.

[0070] In other words, this embodiment proposes a security risk attention-oriented large-scale violation semantic detection model from two aspects: text semantic information and text temporal information. This model aims to fully mine textual information that contributes significantly to semantic matching from both word and sentence dimensions using BERT, including surface features, syntactic features, and semantic features. It combines sentence temporal information and utilizes the GAT network to improve algorithm performance. The hidden state matrices and attention matrices of multiple intermediate layers of BERT are used as the node feature matrices and adjacency matrices for the corresponding number of GATs, respectively. A weighting strategy is employed to weight different GAT layers. A security risk attention and violation hybrid loss function based on a violation lexicon are designed to drive the model to pay more attention to domain-specific violation keywords during training, thereby achieving the identification of violation content and overcoming the shortcomings of existing semantic detection methods based on simple embedding vector similarity calculations.

[0071] Preferably, the large model violation semantic detection method of this embodiment includes the following steps:

[0072] Step S1: Use BERT to obtain multiple levels of text semantic features;

[0073] Step S2: Use GRU to obtain the temporal features of the text;

[0074] Step S3: Construct a violation word library and design a security risk attention mechanism based on the violation word library.

[0075] Step S4: Concatenate the hidden state matrix, temporal features, and security risk features of the BERT target layer to obtain a combined feature matrix;

[0076] Step S5: Weight the attention state matrix of the BERT target layer according to the importance of text hierarchical information to obtain the weighted attention matrix;

[0077] Step S6: Input the combined feature matrix and attention matrix into the corresponding GAT layer, and fuse the outputs of each GAT layer according to the attention-normalized weights.

[0078] Step S7: Design a hybrid loss function based on a violation word library and train the model.

[0079] Specifically, the above steps are explained in detail below.

[0080] Step S1: Use BERT to obtain multiple levels of text semantic features.

[0081] BERT is a pre-trained language model based on the Transformer encoder, with bidirectional semantic understanding at its core. BERT's multi-layered Transformer encoder achieves progressive abstraction of semantic features through self-attention mechanisms and feedforward neural networks. It has multiple encoder layers, each generating an intermediate hidden state, and the model's final output is obtained by combining the hidden states of all the encoder layers. In each encoder layer, the self-attention mechanism calculates the association weights between each token and all other tokens in the sentence, aggregating contextual information; the feedforward neural network then performs a non-linear transformation on the aggregated features.

[0082] Based on existing research, the semantic features captured at each level differ, including lexical, syntactic, and semantic information. In summary, each level of text feature emphasizes different aspects of sentence understanding: lower-level features are suitable for tasks relying on local details, such as part-of-speech tagging and named entity recognition; mid-level features can support phrase similarity calculation and syntactic analysis; and high-level features are suitable for scenarios requiring global understanding, such as sentence classification and semantic similarity determination.

[0083] The first stage is the encoding phase. Let the sentence from the training phase be... After word segmentation, a [CLS] tag is added at the beginning of the sentence to aggregate sentence-level features, and a [SEP] tag is added at the end of the sentence to distinguish sentence boundaries. Subsequently, each token is mapped to a pre-trained word embedding (containing the semantics of the word itself), and positional embeddings (marking the position information of each token in the sentence) and segment embeddings (distinguishing different sentences in sentence pair tasks) are superimposed to form an initial 768-dimensional (12-layer BERT) input vector matrix, providing the underlying semantic foundation for subsequent encoding. Therefore, the input to BERT is ([CLS)). [SEP]), For the first An embedded representation of a character.

[0084] Next comes the multi-layer Transformer encoding stage. Taking the basic BERT version (12 layers) as an example, as the number of layers increases, features gradually progress from the surface to the deeper layers: the bottom layers (layers 1-4) focus more on local semantics, such as the literal meaning of words, parts of speech, and short-distance collocations; the middle layers (layers 5-8) capture phrase-level or syntactic structure features, or the subject-verb-object logic of sentences; the top layers (layers 9-12) focus on the global semantics of sentences, and the high-level outputs of [CLS] tags become a condensed representation of the overall semantics of the sentence. Let the hidden state matrix of BERT be... The attention matrix is ,in It is BERT's first The hidden state matrix of the layer, It is the first Attention matrix of the layer.

[0085] Finally, feature layer parsing is performed. By extracting the output vectors from different layers, multi-level semantic features can be obtained. Considering that the task of this embodiment is semantic detection and based on existing research results, this embodiment uses the hidden state matrices and attention matrices of layers 1, 4, 5, 8, 9, and 12 as inputs for the next layer to fuse local and global information and further enhance the richness of semantic representation.

[0086] Step S2: Use GRU to obtain the temporal features of the text.

[0087] Previous research has found that adding a temporal model after BERT can learn the potential relationships between individual characters in the input and the long-sequence temporal features of the text, which has a positive effect on text semantic judgment. Therefore, considering the timeliness issue of temporal models in processing long sequences, the text embedding vectors obtained in this embodiment are processed by a GRU to learn the contextual features of the text. Let the sentence... The time series feature matrix is .

[0088] Step S3: Construct a violation word database and design a security risk attention mechanism based on the violation word database.

[0089] First, a violation terminology database is constructed. The initial database sources include general standards, domain-specific regulations, and historical violation data. Regarding general standards, this embodiment is based on a general regulatory framework, with boundaries refined to reflect domain characteristics. Specifically, it is based on the five major categories and 32 subcategories of security risk dimensions (violation of values, infringement of rights, commercial violations, etc.) of the "Basic Requirements for Security of Generative Artificial Intelligence Services" (TC260-003) for corpora and generated content, forming the foundation of the keyword database. Regarding domain-specific regulations, specific regulations for each application domain are deeply explored to extract professional violation terms. Regarding historical violation data, based on the compliance framework, in-depth mining of historical violation data supplements the core terms with derivatives, variants, and vivid expressions, addressing the issue of strong principles but insufficient details in general and domain-specific regulations. The collection of historical violation data should be divided into two categories: first, publicly available historical case data, including violation notices issued by regulatory authorities, illegal expressions in judicial judgments, and violation case databases published by industry associations, from which frequently occurring derivative terms are extracted; second, historical violation data within application scenarios, such as user report records on social media platforms and manually reviewed and marked violation texts, capturing unique, implicit expressions and variations specific to each scenario. It is worth noting that to ensure the accuracy and timeliness of the thesaurus, a mechanism for both regular and irregular thesaurus updates needs to be established.

[0090] Secondly, a security risk attention mechanism is designed. To enable the model to learn more about text violations, based on a violation word list, the model obtains violation attention by querying whether the input word is a sentiment violation word. First, the input character is judged by querying a sentiment word dictionary. Whether it is a prohibited word, a random one is generated. Feature vectors of the same dimension , so that each input character They all have a feature vector It is worth noting that vectors It includes two types: illegal embeddings and non-illegal embeddings. The feature values ​​of both types are fixed after initialization. In other words, by querying the illegal word library, each input character will have a feature vector with fixed values ​​and dimensions. Definition For input characters The safety risk attention score is then...

[0091]

[0092]

[0093] in, express and The score feedback mapping between them; For the first Embedded representation of a single character; superscript T This represents the matrix transpose operation; , , and All of these are learnable parameters. Therefore, the normalized security risk attention score is...

[0094]

[0095] in, Indicates input character The normalized security risk attention distribution. Then the security risk vector is...

[0096]

[0097] in, Indicates the encoder at time step The security risk vector; The number of text characters. Transform the security risk feature vector into a security risk feature matrix of appropriate dimensions. .

[0098] Step S4: Concatenate the hidden state matrix, temporal features, and security risk features of the BERT target layer to obtain a combined feature matrix.

[0099] In this embodiment, the hidden state matrices of BERT layers 1, 4, 5, 8, 9, and 12 are concatenated with the GRU temporal feature matrix as the input to the next layer, GAT. The hidden state matrices of layers 1, 4, 5, 8, 9, and 12 are concatenated with the temporal feature matrix obtained in step S2 and the security risk feature matrix obtained in step S3, respectively, resulting in the combined state matrix as follows: .

[0100] Step S5: Weight the attention state matrix of the BERT target layer according to the importance of the text hierarchical information to obtain the weighted attention matrix.

[0101] In this embodiment, the hidden state matrices of BERT layers 1, 4, 5, 8, 9, and 12 are selected, and the BERT attention matrix is ​​defined as follows: GAT is a graph neural network based on an attention mechanism, the core of which is to assign differentiated weights to the neighbors of nodes in the graph. In this embodiment, the hidden state matrices and attention matrices of BERT layers 1, 4, 5, 8, 9, and 12 are used as the node feature matrices and adjacency matrices of GAT, respectively, to improve similarity recognition performance. It is worth noting that since the task of this embodiment is semantic detection, focusing more on the semantic understanding of text, based on existing research conclusions, the GAT adjacency matrix in this embodiment is obtained by setting the weight coefficients of the attention matrices at each level of BERT according to the logic of lexical < syntactic < semantic, i.e., ...

[0102]

[0103] in, It is the first The adjacency matrix of GATs. This indicates the importance of different layers in BERT; , , .

[0104] Step S6: Input the combined feature matrix and attention matrix into the corresponding GAT layer, and fuse the outputs of each GAT layer according to the attention-normalized weights.

[0105] The specific operation of inputting the combined feature matrix and attention matrix into the corresponding GAT layer is to concatenate the temporal features from BERT layers 1, 4, 5, 8, 9, and 12 into a combined state matrix. With the weighted attention matrix This serves as the node feature matrix and adjacency matrix corresponding to GAT. In this embodiment, there are... The BERT information output at each level corresponds to... One GAT layer, ,layer The corresponding weight is , Let the first... The normalized weights of the GAT layers are: Then there is

[0106]

[0107] The final output of the GAT layer is linearly weighted according to the normalized weights to obtain the final sequence output, i.e.,

[0108]

[0109] in, This is the final output of the model; For the first The output of each GAT layer.

[0110] Step S7: Design a hybrid loss function based on a violation word library and train the model.

[0111] The loss function is a quantization guide in the model training process, used to measure the distance between the model's actual output and the true label. Since the downstream task in this embodiment is to identify illegal semantics, it is necessary to train the semantic recognition model guided by the illegal sensitive words constructed in the rule layer.

[0112] Based on this, this embodiment proposes a hybrid loss function that includes a violation binary attention layer. The loss function of the BERT model GAT layer loss function and the binary loss function for violations It is composed of three parts in a linear weighted combination.

[0113] Generally, determining whether a sentence contains prohibited content is based on the presence of prohibited words. In other words, the model should be trained with the goal of identifying and capturing information about prohibited words in the source text. Therefore, building upon existing research, this embodiment introduces a binary attention layer to learn the positional distribution information of prohibited words in the input text. The ultimate goal of the binary attention layer is to obtain the probability of each word in the source text being a prohibited word, independent of other words, rather than the normalized probability distribution of all words in the source text. Therefore, the sigmoid function is used instead of the commonly used softmax function in the binary attention layer. The binary attention score and the corresponding binary attention probability distribution are then:

[0114]

[0115]

[0116] in, equal , indicating the first Input characters The binary attention score; Indicates the first Input characters It is the distribution of attention to the prohibited words; Let represent the sigmoid function; the definitions of other parameters are the same as before. Then, the binary attention term can be obtained as follows:

[0117]

[0118] in, It is the violation loss term of the violation cross-entropy; It is a binary cross-entropy function, which can be directly imported from PyTorch; It is a sequence with input A binary attention vector of equal length, if the input character If it is a prohibited word, then the corresponding bit of that character will be removed. Set to 1 if it is not a violation, and set to 0 otherwise. In this embodiment, a violation word database is constructed to determine whether a word is a violation word.

[0119] Loss function of BERT layer Defined as

[0120]

[0121] in, The number of samples; Number of categories; For the sample Does it belong to a category? The predicted score; This is the model's predicted output.

[0122] The loss function of the GAT layer is

[0123]

[0124] in, For the model to sample The predicted value of the true category; For the model to sample Predicted values ​​for other categories.

[0125] Finally, by weighting and combining the above formulas, the final violation mixture loss function of the model is:

[0126]

[0127] in, These are hyperparameters that need to be determined through preliminary experiments; This is the final form of the violation hybrid loss function proposed in this embodiment, which can drive the model to effectively capture violation information in the source text.

[0128] Example 3

[0129] This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement a large-model semantic violation detection method of embodiment 1 or 2. The computer program can be in the form of source code, object code, executable file, or some intermediate form.

[0130] Example 4

[0131] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a large-model semantic violation detection method of Embodiment 1 or 2. The computer program can be in the form of source code, object code, executable file, or some intermediate form. The storage medium includes any entity or device capable of carrying computer program code, a recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0132] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

[0133] It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

Claims

1. A method for detecting semantic violations in large models, characterized in that, include: Multiple levels of text semantic features are obtained by representing the transformation model through a bidirectional encoder, and text temporal features are obtained through a gated recurrent unit. Construct a violation word library and set up a security risk attention mechanism based on the violation word library to obtain a security risk feature matrix; The hidden state matrix of the target layer of the bidirectional encoder representation transformation model, the text temporal features, and the security risk feature matrix are concatenated to obtain a combined feature matrix. The attention matrix of the target layer of the bidirectional encoder representation transformation model is weighted according to the importance of text hierarchical information to obtain the weighted attention matrix; The combined feature matrix and the weighted attention matrix are input into the corresponding graph attention network. The outputs of each layer of the graph attention network are fused according to the attention-normalized weights to obtain the fused output. A violation semantic detection model is trained by a violation hybrid loss function based on the violation lexicon. The violation semantic detection model includes a bidirectional encoder representation transformation model, a gated recurrent unit, and a graph attention network. And perform large-scale semantic violation detection using a pre-trained semantic violation detection model; The security risk attention mechanism based on the violation word library includes: querying whether the input character is a violation word in the violation word library, and assigning a corresponding feature vector to each input character; calculating a security risk attention score based on the input character and its corresponding feature vector, normalizing the security risk attention score to obtain a normalized security risk attention distribution; calculating a security risk vector based on the normalized security risk attention distribution and the input character, and converting the security risk vector into a security risk feature matrix.

2. The method for detecting semantic violations in a large model according to claim 1, characterized in that, Before obtaining the semantic features of the text, the sentences in the training phase are segmented, including: adding a tag at the beginning of the sentence to aggregate sentence-level features, adding a tag at the end of the sentence to distinguish sentence boundaries, mapping each segmented word to a word embedding, and superimposing position embedding and segment embedding to form an initial input vector matrix.

3. The method for detecting semantic violations in a large model according to claim 1, characterized in that, The initial sources of the violation term database include general specifications, domain regulations and systems, and historical violation data. The historical violation data includes publicly available historical case data, historical violation data within application scenarios, and expanded historical violation data. The method for generating the expanded historical violation data includes: supplementing the core terms with derivative terms, variant terms, and vivid expressions through in-depth mining of historical violation data based on the compliance framework.

4. The method for detecting semantic violations in a large model according to claim 1, characterized in that, The step of weighting the attention matrix of the target layer of the bidirectional encoder representation transformation model according to the importance of text-level information includes: setting the weight coefficients of the attention matrix of each target layer according to the logic of lexical information importance < syntactic information importance < semantic information importance, with the sum of all weight coefficients being 1, and the value range of each weight coefficient being 0 to 1.

5. The method for detecting semantic violations in a large model according to claim 1, characterized in that, The step of inputting the combined feature matrix and the weighted attention matrix into the corresponding graph attention network includes: the number of layers in the graph attention network is consistent with the number of target layers in the bidirectional encoder representation transformation model, and each layer of the graph attention network corresponds to a set of combined feature matrices and weighted attention matrices of the target layers.

6. The method for detecting semantic violations in a large model according to claim 1, characterized in that, The violation hybrid loss function includes a bidirectional encoder representation transformation model loss function, a graph attention network loss function, and a violation binary loss function. The violation binary loss function is constructed based on the binary cross-entropy function and is used to learn the positional distribution information of violation words in the input text. The input contains a binary attention vector of the same length as the text sequence. The binary attention vector is used to label whether the input characters are violation words according to the violation word library.

7. A large-scale model semantic violation detection system, characterized in that, include: The feature extraction module is configured to obtain multiple different levels of text semantic features through a bidirectional encoder to represent the transformation model, and to obtain text temporal features through a gated recurrent unit. Construct a violation word library and set up a security risk attention mechanism based on the violation word library to obtain a security risk feature matrix; The feature concatenation module is configured to concatenate the hidden state matrix of the target layer of the bidirectional encoder representation transformation model, the text temporal features, and the security risk feature matrix to obtain a combined feature matrix. The attention matrix of the target layer of the bidirectional encoder representation transformation model is weighted according to the importance of text hierarchical information to obtain the weighted attention matrix; The feature fusion module is configured to input the combined feature matrix and the weighted attention matrix into the corresponding graph attention network, and fuse the outputs of each layer of the graph attention network according to the attention-normalized weights to obtain a fused output. The model training and application module is configured to train a violation semantic detection model using a violation hybrid loss function based on the violation lexicon. The violation semantic detection model includes a bidirectional encoder representation transformation model, a gated recurrent unit, and a graph attention network. And perform large-scale semantic violation detection using a pre-trained semantic violation detection model; The security risk attention mechanism based on the violation word library includes: querying whether the input character is a violation word in the violation word library, and assigning a corresponding feature vector to each input character; calculating a security risk attention score based on the input character and its corresponding feature vector, normalizing the security risk attention score to obtain a normalized security risk attention distribution; calculating a security risk vector based on the normalized security risk attention distribution and the input character, and converting the security risk vector into a security risk feature matrix.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the large model violation semantic detection method according to any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the large model violation semantic detection method according to any one of claims 1-6.

Images