A language steganalysis method based on knowledge injection and knowledge encoding

Abstract

The present invention discloses a language steganalysis method based on knowledge injection and knowledge encoding, which belongs to the field of information hiding. First, a language steganalysis system model based on knowledge injection and knowledge encoding is constructed. The system model preprocesses the input sentence with the help of a knowledge graph to complete the knowledge injection. Then, BERT is used on the sentence tree and the visible matrix to complete the extraction of semantic features and perform knowledge injection semantic encoding. During the preprocessing, a KE module of graph structure modeling is used to extract knowledge from the input sentence and the knowledge graph to obtain a mini-graph, and the mini-graph is knowledge encoded by GAT. Finally, the captured entity node features and the representation vector obtained from the knowledge injection module are connected in series and added to a fully connected layer. The system model predicts whether the input text is a normal carrier or a steganographic carrier through the output of the fully connected layer. The present invention solves the fragmentation problem of network text and makes the results of language steganalysis more accurate.

Description

Technical Field

[0001] The present invention belongs to the field of information hiding, relates to text steganalysis detection, and particularly relates to a language steganalysis method based on knowledge injection and knowledge encoding. Background Art

[0002] Text steganalysis has rapidly advanced with advances in natural language processing technology. Despite significant improvements in detection capabilities, achieving satisfactory results in real-world online text detection scenarios remains difficult. A significant reason for this is that text in public online spaces, such as Twitter, is often highly fragmented, with individual texts typically possessing limited indicative features. Current detection models face challenges extracting sufficiently rich features from individual input texts, hindering a comprehensive understanding of the text and limiting the model's ultimate detection capabilities.

[0003] Yang et al. [Sesy: Linguistic steganalysis framework integrating semantic and syntactic features] proposed a Sesy scheme that uses a powerful pre-trained language model, Bidirectional Encoder Representation Transform (BERT), to extract semantic features. They also designed a Sesy framework to account for grammatical changes caused by the embedding of secret information, effectively improving existing state-of-the-art linguistic steganalysis algorithms. This scheme enhances the capabilities of text steganalysis to a certain extent, but these detection models face severe challenges in real-world network detection scenarios due to the fragmented nature of online text.

[0004] Fu et al. [Hga: Hierarchical Feature Extraction with Graph and Attention Mechanism for Linguistic Steganalysis] proposed a scheme called Hga, which introduced LS-BGAT. This innovative approach merges text nodes with word nodes to form a large heterogeneous graph and uses a graph attention network (GAT) to fully understand the relationships between words and the corpus. However, this scheme still fails to effectively address the text fragmentation challenge faced in online steganalysis. Achieving a deep understanding of text based on a single sentence input is nearly impossible. Summary of the Invention

[0005] The present invention proposes a language steganalysis method based on knowledge injection and knowledge encoding to solve practical challenges of text steganalysis caused by network text fragmentation.

[0006] The language steganalysis method based on knowledge injection and knowledge encoding has the following specific steps:

[0007] Step 1: Build a language steganalysis system model based on knowledge injection and knowledge encoding;

[0008] The system model is named KIKE, which includes two parts: semantic encoding based on knowledge injection and knowledge encoding based on knowledge extraction.

[0009] Step 2: The language steganalysis system model preprocesses the input sentence with the help of the knowledge graph to adapt to the existing sequential language model framework and complete knowledge injection;

[0010] The specific preprocessing process is:

[0011] Step 201: The input sentence S is a set of tokens, represented by S = {w1, w2, ..., w n}, where n is the length of the sentence, w i is the i-th token of the sentence. The knowledge graph K is a graph structure consisting of triples of entities and relationships between entities, denoted as K(V,E). K contains multiple triples (v i ,e k ,v j ), where v i and v j is an entity, e k Represents relationships between entities.

[0012] Step 202: Use the knowledge graph to convert the input sentence from its original form S to a sentence tree S':

[0013]

[0014] Among them, (w i ,...,w i+l-1 ) represents an entity E consisting of l words i , Indicates connection to E i Multiple branches.

[0015] Step 203: renumber the tokens in the sentence tree S' and perform position embedding of the language model using a combination of soft position index and hard position index;

[0016] The specific implementation process is:

[0017] First, the soft position index of each token in the original sentence is 1 to n. If the distance between the content injected into the knowledge graph and the soft position i of the original sentence is x, then the soft position index of the content injected into the knowledge graph is i+x, and is mapped to the same position vector in the position embedding process;

[0018] The hard position index is then used to reorder tokens in the sentence tree and flatten the sentence tree into a linear sequence for the encoding stage.

[0019] Finally, the index of each token in the sentence tree is obtained by combining the above soft and hard position indexes.

[0020] Step 204: construct a visibility matrix M using hard position indexes to guide the visibility range of the knowledge branch;

[0021] It can be seen that the matrix M is expressed as follows:

[0022]

[0023] in, Indicates that the two tags are on the same branch, Indicates that the two markers are not on the same branch. i and j are hard position indices.

[0024] Step 3: Use BERT on the sentence tree and visibility matrix to extract semantic features and perform knowledge injection semantic encoding;

[0025] The specific process is:

[0026] In step 301, the embedding layer of BERT includes word embedding, position embedding, and segment embedding. The position embedding part uses a soft position index to represent the relative position of the token in the sentence. The word embedding and segment embedding parts retain the original mapping method.

[0027] Step 302: embed these three pieces of information into a high-dimensional feature space and add them together to obtain hidden layer variables H0 = {h1, h2, ..., h n ′}∈R n′×d , where n' is the length of the flattened sequence after compressing the sentence tree, and d is the number of features.

[0028] In step 303, H0 is passed to the Mask-Transformer module, which consists of L stacked Mask-Transformer sublayers. The input vector set of each layer is the output vector set of the previous layer, as shown below:

[0029] H l =Mask-Transformer l (H l-1 ),1≤l≤L

[0030] Each Mask-Transformer sublayer consists of a multi-head attention layer, a fully connected layer, and two residual connection normalization layers. The visibility matrix is ​​used to control the correlation calculation between tokens during the attention calculation process. Each hidden vector group (Q l,K l ,V l ) is calculated as follows:

[0031]

[0032] Among them, W q 、W k and W v is a trainable parameter matrix, h l is the combination of all hidden vectors in layer l. If vector h l i For h l j is visible, then the correlation calculation is the same as in BERT. If h l i and h l j is invisible, because M ij =-∞, attention score Will be set to 0. S l is the attention score of layer l.

[0033] In step 304, after refining the L-layer semantic features, a sigmoid function is used to obtain the representation vector of the knowledge injection semantic encoding:

[0034]

[0035] Represents the hidden layer variable output by the L-th Mask-Transformer sublayer;

[0036] Step 4: During preprocessing, the KE module based on graph structure modeling is used to extract knowledge from the input sentence and knowledge graph to obtain a mini-graph.

[0037] First, we extract the recognized entities from the input sentence and connect them one by one, and then connect them to the corresponding object branches in the knowledge graph to form a mini-graph G(γ,E), where γ={v1,…,v m}, m is the number of entities in the sentence tree, E = {ε ab =(v a ,v b )1≤a,b≤m} is an edge set.

[0038] H is converted into C The entity vector in is used as the initial vector of the graph node, and an adjacency matrix A∈R is used m×m To represent the connection relationship between entities.

[0039] Step 5: Encode the knowledge of the mini-graph through a two-layer GAT.

[0040] The specific implementation process is:

[0041] First, the first layer uses a multi-head graph attention layer to extract the correlation information between nodes. For each multi-head graph attention layer, a globally shared weight matrix W is used to perform a linear transformation on the initial vector.

[0042] Then, the Softmax method is used to calculate the graph attention coefficient α between each node and its connected nodes ij , the specific expression is as follows:

[0043]

[0044] Among them, N i Represents the node e i The set of all connected neighbor nodes, W∈R d×d' and is a trainable model parameter, and LeakyReLU is a nonlinear activation function.

[0045] Finally, using α ij , the feature representation of node i is updated by adaptively aggregating information from neighboring nodes as follows:

[0046]

[0047] Where σ is the sigmoid activation function. The output vectors of multiple GALs are concatenated and fed into the next separate GAL to obtain a vector set e of m nodes 2 .

[0048] Step 6: After knowledge encoding is completed, the captured entity node features and the representation vector obtained from the knowledge injection module are concatenated and added to a fully connected layer. The system model predicts whether the input text is a normal carrier or a steganographic carrier through the output of the fully connected layer.

[0049] Step seven: Train the prediction accuracy of the system model. During the training phase, the cross entropy loss function is used to update the system model parameters. When the loss function is minimized, the training is completed.

[0050] The cross entropy loss function is:

[0051]

[0052] Among them, y is the label of the normal vector, is the predicted label output by the model, and N is the number of input samples.

[0053] The advantages of the present invention are:

[0054] 1. This paper proposes a language steganalysis method based on knowledge injection and knowledge encoding to address the fragmentation of online text in real online environments. This method can be implemented through the semantic extraction, cognitive verification, and information fusion components of knowledge injection. By introducing domain knowledge through knowledge graphs, language models can better assist in classification tasks.

[0055] 2. To achieve a novel language steganalysis method based on knowledge injection and knowledge encoding, this invention uses a domain-specific knowledge graph to inject triples into sentences. The resulting sentence tree and visibility matrix are then combined with the existing Transformer architecture to extract semantic features. In knowledge encoding, entities are integrated into mini-graphs and cognitive verification is performed using graph neural networks. The combination of these two processes results in more accurate language steganalysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a schematic diagram of the language steganalysis system model based on knowledge injection and knowledge encoding proposed by the present invention;

[0057] Figure 2 This figure shows the effect of inserting different numbers of branches on the detection effect F1 value in the embodiment. DETAILED DESCRIPTION

[0058] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0059] Inspired by the innovative application of a new language steganalysis framework leveraging external knowledge, this paper focuses on designing a more powerful text feature extraction model using most text steganalysis algorithms to achieve efficient steganalysis. However, in real online environments, they face significant challenges related to the fragmentation of online text, which greatly limits their practical applicability.

[0060] This paper proposes a language steganalysis method based on knowledge injection and knowledge encoding, named KIKE. The core idea of ​​the KIKE model is to introduce external knowledge to guide the fine-tuning of entity vectors at both the data and model levels. This is equivalent to annotating entities in text to help the model better understand the text.

[0061] In knowledge injection, triples are injected into sentences using knowledge graphs from professional domains. The generated sentence tree and visibility matrix are combined with the existing Transformer architecture to extract semantic features. In knowledge encoding, entities are integrated into mini-graphs and cognitive verification is performed using graph neural networks. The protocol flow of each stage is as follows: Figure 1The proposed algorithm can be divided into two types: semantic encoding algorithm based on knowledge injection and knowledge encoding algorithm based on knowledge extraction. Knowledge extraction occurs in the preprocessing stage, and the semantic encoding part integrates external information.

[0062] The specific steps are as follows:

[0063] Step 1: Build a language steganalysis system model based on knowledge injection and knowledge encoding;

[0064] The system model includes the semantic extraction part of knowledge injection, the cognitive verification part and the information fusion part. By introducing professional domain knowledge with the help of knowledge graph, the language model can better assist in performing classification tasks. Figure 1 As shown in Figure 3, the proposed system model is divided into two parts: semantic encoding based on knowledge injection at the bottom and knowledge encoding based on knowledge extraction at the top.

[0065] Semantic encoding based on knowledge injection: Triples are injected into sentences with the help of knowledge graphs from professional fields, and the generated sentence tree and visibility matrix are combined with the existing Transformer architecture to extract semantic features.

[0066] Knowledge encoding based on knowledge extraction: responsible for integrating entities into mini-graphs and using graph neural networks for cognitive verification.

[0067] Step 2: In order to better integrate knowledge into sentences and adapt to the existing sequential language model framework, the system model preprocesses the input sentences with the help of the knowledge graph before extracting semantic features, which is called "knowledge injection";

[0068] The specific preprocessing process is:

[0069] Step 201: The input sentence is represented as S = {w1, w2, ..., w n}, where n is the length of the sentence, w i is the i-th token of the sentence. The knowledge graph is represented as K(V,E), where K contains multiple triples (v i ,e k ,v j ), where v i and v j is an entity, e k Represents relationships between entities.

[0070] In this step, the input sentence and knowledge graph are represented as data structures. Sentence S is a set of tokens, each representing a word or identifier in the sentence. The knowledge graph K is a graph structure consisting of triplets of entities and their relationships. This representation facilitates subsequent processing and analysis.

[0071] Step 202: Use the knowledge graph to convert the input sentence from its original form S to a sentence tree S':

[0072]

[0073] Among them, (w i ,...,w i+l-1 ) represents an entity E consisting of l words i , Indicates connection to E i Multiple branches.

[0074] In this step, the original sentence S is transformed into a more structured form, where the entity E i It contains a series of related words and is connected to other entities through branches to form a sentence tree structure. Such a transformation helps capture the relevance and structural information between words in a sentence.

[0075] Step 203: Since the linear sequence is converted into a tree structure, the tokens in S' need to be renumbered to adapt to the position embedding and attention calculation mechanism of the advanced language model. The embedding is performed using a combination of soft position indexing and hard position indexing. The soft position indexing is used to ensure that the context structure information is not affected during the position embedding process.

[0076] Figure 1 In the embodiment shown, the input sentence is "We use syntactic tree pruning strategy to optimize semantic space". The specific implementation process of soft position indexing and hard position indexing in the position embedding process is as follows:

[0077] Calculation of soft position index: For each token in the sentence tree, calculate the soft position index. For example, for "graph" and "information", their distance from "pruning" in the sentence tree is 2, so their soft position index is 7. These soft position indexes will be used to map to the corresponding position vectors in the position embedding process.

[0078] Calculation of hard position index: The hard position index is used to reorder the tokens in the sentence tree and flatten the sentence tree into a linear sequence for the subsequent encoding stage. For example, through the hard position index, the index of "graph" is 7, while the index of "information" is 10.

[0079] The combination of soft and hard position indexing ensures that the conversion process not only takes into account the structural information of the sentence tree but also adapts to the language model's requirements for position embedding and attention calculation. This helps preserve contextual information and improves the model's understanding and processing capabilities of input data.

[0080] In step 204, since the knowledge injection method introduces external tags into the representation learning process, it is necessary to ensure that the knowledge branches are only used to enrich the meaning of their corresponding entities and avoid interfering with the semantic representation of other words in the original sentence. Therefore, the visibility matrix M is constructed using hard position indexing as shown below:

[0081]

[0082] in, Indicates that the two tags are on the same branch, and Indicates that they are not in the same branch. i and j are hard position indices.

[0083] The visibility matrix M constructed in this way guides the visibility range of the knowledge branch through hard position indexing, ensuring that the injected knowledge only affects the corresponding entity without disrupting the semantic representation of other words in the original sentence. This is an important step in maintaining the robustness and understanding of the model.

[0084] Step 3: To explore the rich meanings of words from multiple perspectives and capture long-range dependencies between characters, BERT is used in practice to extract semantic features.

[0085] BERT's embedding layer consists of three parts: word embedding, position embedding, and segment embedding. The embedding process is:

[0086] First, a soft position index is used in the position embedding part to represent the relative position of the token in the sentence. The other two parts retain the original mapping method.

[0087] Then, after embedding these three types of information into the high-dimensional feature space, they are added together to obtain the hidden layer variables H0 = {h1,h2,...,h n′}∈R n′×d , where n' is the length of the flattened sequence after compressing the sentence tree, and d is the number of features.

[0088] Finally, H0 is passed to the Mask-Transformer module, which consists of L stacked Mask-Transformer sublayers. The input vector set of each layer is the output vector set of the previous layer, as shown below:

[0089] H l =Mask-Transformer l (H l-1 ),1≤l≤L

[0090] The calculation of each layer includes a multi-head attention layer, a fully connected layer, and two residual connection normalization layers. Unlike the BERT model, this paper uses a visible matrix to control the correlation calculation between tokens during the attention calculation process. Formally, the calculation of each hidden vector group is as follows:

[0091]

[0092] Among them, W q 、W k and W v is a trainable parameter matrix, h l is the combination of all hidden vectors in the lth layer. As shown in the formula, if the vector h l i For h l j is visible, then the correlation calculation is the same as in BERT. If h l i and h l j is invisible, because M ij =-∞, attention score will be set to 0.

[0093] After L-layer semantic feature refinement, the representation vector of knowledge-injected semantic encoding is obtained by using the Sigmoid function:

[0094]

[0095] Word embedding: Use the word embedding part of the BERT model and retain the original mapping method.

[0096] Position Embedding: In the position embedding part, soft position index is used to represent the relative position of tokens in the sentence to preserve the order information between tokens.

[0097] Segment Embedding: Keep the original segment embedding mapping method.

[0098] Mask-Transformer Module: H0 is passed to the Mask-Transformer module, which consists of L stacked Mask-Transformer sublayers. The input vector set of each layer is the output vector set of the previous layer. The calculation of each layer includes a multi-head attention layer, a fully connected layer, and two residual connection normalization layers. The visibility matrix is ​​used to control the correlation calculation between tokens during the attention calculation.

[0099] Sigmoid function: After L layers of semantic feature refinement, the representation vector of the knowledge-injected semantic encoding is obtained by using the Sigmoid function.

[0100] This process aims to capture the rich meaning of words from multiple perspectives while considering the long-distance dependencies between characters. A visible matrix is ​​introduced through the Mask-Transformer module to control the relevance of attention calculations.

[0101] In step 4, the semantic encoding part integrates external information and can be independently input into the discriminator to complete the detection task. However, in order to better explore the potential relationships between entity concepts from a cognitive perspective and identify the knowledge consistency in the content from a cognitive perspective, this paper designs a KE module using graph structure modeling, which includes two parts: knowledge extraction and knowledge encoding.

[0102] Knowledge extraction occurs in the preprocessing stage, extracting recognized entities from the sentence and connecting them one by one, and then connecting them to the corresponding object branches in the knowledge graph to form a small graph G(γ,E), where γ={v1,…,v m}, m is the number of entities in the sentence tree, E = {ε ab =(v a ,v b )1≤a,b≤m} is an edge set.

[0103] H is converted into C The entity vector in is used as the initial vector of the graph node, and an adjacency matrix A∈R is used m×m To represent the connection relationship between entities.

[0104] This part aims to use graph structures to model the relationships between entity concepts. By extracting knowledge from the resulting small graphs, we can better identify knowledge consistency within the content. This knowledge encoding process is performed using a graph neural network (GNN) to further capture the semantic relationships between entities. This helps to better understand entity associations in text from a cognitive perspective.

[0105] In step 5, a two-layer GAT (Graph Attention Network) is applied to encode the mini-graph. This network can simultaneously extract the local relationships between text nodes and the global structure of the entire graph, and infer similar structures and features from the learned graph data, showing good generalization performance.

[0106] The specific implementation process is:

[0107] First, the first layer uses a multi-head graph attention layer to explore the association information between nodes from different perspectives. For each multi-head graph attention layer, a globally shared weight matrix W is used to linearly transform the initial vector.

[0108] Then, the Softmax method is used to calculate the graph attention coefficient α between each node and its connected nodes ij , the specific expression is as follows:

[0109]

[0110] Among them, N i Represents the node e i The set of all connected neighbor nodes, W∈R d×d' and a∈R 2d' is a trainable model parameter, and LeakyReLU is a nonlinear activation function.

[0111] Using α ij , the feature representation of node i can be updated by adaptively aggregating information from neighboring nodes as follows:

[0112]

[0113] Where σ is the sigmoid activation function. The output vectors of multiple GALs are concatenated and then fed into the next separate GAL (Graph Attentional Layer) to obtain the vector set e of m nodes 2 .

[0114] The purpose of this entire step is to capture the complex relationships between text nodes through a multi-head graph attention mechanism, fully utilizing the information in the graph structure to obtain a higher-level semantic representation. This helps the model better understand the associations between entities and improves its ability to understand and express text.

[0115] Step 6. Once the knowledge encoding is completed, the captured entity node features and the representation vector obtained from the knowledge injection module are concatenated and added into a fully connected layer to determine whether the text is a stego carrier.

[0116] During the training phase, the model parameters are updated using the following cross entropy loss function:

[0117]

[0118] Among them, y is the label of the normal vector, is the predicted label output by the model, and N is the number of input samples.

[0119] Information Fusion: Once knowledge encoding is completed, i.e., entity node features and representation vectors obtained from the knowledge injection module, these two pieces of information are concatenated. This is to fully utilize the different levels and types of information obtained through the knowledge injection and knowledge encoding stages to improve the model's overall understanding of the text.

[0120] Fully Connected Layer: The concatenated feature vectors are fed into a fully connected layer. A fully connected layer is a common neural network layer used to learn complex relationships between features. Here, it further combines and transforms the input features to prepare for the final classification task.

[0121] Text Detection Task: The final fully connected layer output is used to determine whether the text is a stegoscopy carrier. This is achieved through the model's binary classification output: the model predicts whether the input text is a normal carrier or a stegoscopy carrier. The goal of this task is to improve the model's text detection performance through parameter updates during training.

[0122] Loss function: During the training phase, the cross-entropy loss function is used as the model's optimization objective. The cross-entropy loss function measures the difference between the probability distribution of the model output and the actual label. By minimizing this difference, the model parameters are adjusted to make the model more accurate in predicting the text category.

[0123] The purpose of this whole step is to combine the information obtained through knowledge injection and knowledge encoding into the final text detection task, thereby improving the model's detection performance for network text steganography.

[0124] This paper analyzes the effectiveness of the proposed scheme and verifies the effectiveness of the proposed language steganalysis scheme based on knowledge injection and knowledge encoding by providing performance analysis.

[0125] Verification 1: The proposed scheme and baseline model were tested for their detection performance on text steganography algorithms for various data sets at different embedding rates. The results are shown in Table 1. In general, the scheme proposed in this invention outperforms all other baseline models. First, thanks to the powerful semantic understanding ability of the BERT model, the detection model based on BERT is significantly better than other deep learning methods, and can improve the accuracy Acc by about 8%. When the FLC embedding amount is 3.98bpw, KIKE's Acc even reaches 98.6%. Secondly, in this scenario, the BERT model alone is usually not so stable in the balance between precision P and recall R. After adopting the Knowledge Infusion strategy, better and more robust results can be achieved.

[0126] Verification 2: Ablation experiments were conducted in Table 2 to quantify the impact of the key components of the proposed model. First, without changing the original structure and parameter size of the BERT model, adding the KI module alone can effectively improve the detection accuracy by about 2% to 3%, proving the effectiveness of the data enhancement strategy of integrating knowledge into sentences. Secondly, by comparing the experimental results of KI and KIKE, it can be found that when bpw is low, the lack of entity relevance in the sentence makes the language model based on data enhancement play a greater role. However, when bpw gradually increases, the added KE module can better detect steganographic traces from a cognitive perspective by checking the consistency of entity semantics.

[0127] Verification 3: A deeper exploration of the number of branches of introduced knowledge was conducted, such as Figure 2 As shown in the figure, KI and KIKE both perform best when the number of branches is 3. This is because appropriate supplementary knowledge helps fully capture the semantics of specialized entities, rather than relying solely on the contribution of individual words within that entity in the language model to influence the meaning of the entire sentence. However, the detection performance decreases when the number of branches increases to 4. This is presumably because too much additional information causes words to deviate from their original position in the word vector space, disrupting the semantic expression of the original sentence.

[0128] During the experiment, the weight of the knowledge encoding input to the classifier was controlled by adjusting the different parameters of the GAT. It was observed that a low number of branches is suitable for knowledge encoding with a smaller number of parameters. This is because a knowledge graph with too few branches requires high precision in the introduced knowledge, and one-sided knowledge may affect the accuracy of the language model. A high number of branches, on the other hand, is suitable for knowledge encoding with a larger number of parameters, allowing the knowledge graph to be more effective. However, an excessive number of branches is not recommended, as judgment still depends on the lexical logic of the sentence itself.

[0129] Table 1 Detection performance of each model for different text steganography algorithms at different embedding rates

[0130]

[0131] Table 2 Comparison of text steganalysis performance of each module in KIKE

[0132]

Claims

1. A language steganalysis method based on knowledge injection and knowledge encoding, characterized in that: The specific steps are as follows: Step 1: Build a language steganalysis system model based on knowledge injection and knowledge encoding; The system model is named KIKE, which includes semantic coding based on knowledge injection and knowledge coding based on knowledge extraction; Step 2: The language steganalysis system model preprocesses the input sentence with the help of the knowledge graph to adapt to the existing sequential language model framework and complete knowledge injection; The specific preprocessing process is: Step 201: The input sentence S is a set of tokens, represented by S = {w1, w2, ..., w n }, where n is the length of the sentence, w i is the i-th token of the sentence; the knowledge graph K is a graph structure consisting of triplets of entities and relationships between entities, denoted as K(V,E), where K contains multiple triplets (v i ,e k ,v j ), where v i and v j is an entity, e k Represents relationships between entities; Step 202: Use the knowledge graph to convert the input sentence from its original form S to a sentence tree S': Among them, (w i ,...,w i+l-1 ) represents an entity E consisting of l words i , Indicates connection to E i Multiple branches of Step 203: renumber the tokens in the sentence tree S' and perform position embedding of the language model using a combination of soft position index and hard position index; Step 204: construct a visibility matrix M using hard position indexes to guide the visibility range of the knowledge branch; It can be seen that the matrix M is expressed as follows: in, Indicates that the two tags are on the same branch, Indicates that the two markers are not in the same branch, i and j are hard position indices; Step 3: Use BERT on the sentence tree and visibility matrix to extract semantic features and perform knowledge injection semantic encoding; The specific process is: Step 301: BERT's embedding layer includes word embedding, position embedding, and segment embedding. Soft position indexing is used in the position embedding to represent the relative position of tokens in the sentence, while the word embedding and segment embedding retain the original mapping method. Step 302: embed these three pieces of information into a high-dimensional feature space and add them together to obtain hidden layer variables H0 = {h1, h2, ..., h n′ }∈R n′×d , where n' is the length of the flattened sequence after compressing the sentence tree, and d is the number of features; In step 303, H0 is passed to the Mask-Transformer module, which consists of L stacked Mask-Transformer sublayers. The input vector set of each layer is the output vector set of the previous layer, as shown below: H l =Mask-Transformer l (H l-1 ),1≤l≤L In step 304, after refining the L-layer semantic features, a sigmoid function is used to obtain the representation vector of the knowledge injection semantic encoding: Among them, H0 L Represents the hidden layer variable output by the L-th Mask-Transformer sublayer; Step 4: During preprocessing, the KE module based on graph structure modeling is used to extract knowledge from the input sentence and knowledge graph to obtain a mini-graph. First, we extract the recognized entities from the input sentence and connect them one by one, and then connect them to the corresponding object branches in the knowledge graph to form a mini-graph G(γ,E), where γ={v1,…,v m }, m is the number of entities in the sentence tree, E = {ε ab =(v a ,v b )1≤a,b≤m} is an edge set; H is converted into C The entity vector in is used as the initial vector of the graph node, and an adjacency matrix A∈R is used m×m To represent the connection relationship between entities; Step 5: Knowledge is encoded in the mini-graph through a two-layer GAT. The captured entity node features and the representation vector obtained from the knowledge injection module are then concatenated and added to a fully connected layer. The system model predicts whether the input text is a normal carrier or a stego carrier based on the output of the fully connected layer. Step 6: Train the prediction accuracy of the system model. During the training phase, the cross entropy loss function is used to update the system model parameters. When the loss function is minimized, the training is completed.

2. The language steganalysis method based on knowledge injection and knowledge encoding according to claim 1 is characterized in that: The specific implementation process of renumbering the tags in the sentence tree S' is: First, the soft position index of each token in the original sentence is 1 to n. If the distance between the content injected into the knowledge graph and the soft position i of the original sentence is x, then the soft position index of the content injected into the knowledge graph is i+x, and is mapped to the same position vector in the position embedding process; The hard position index is then used to reorder the tokens in the sentence tree and flatten the sentence tree into a linear sequence for the encoding stage; Finally, the index of each token in the sentence tree is obtained by combining the above soft and hard position indexes.

3. The language steganalysis method based on knowledge injection and knowledge encoding according to claim 1 is characterized in that: Each Mask-Transformer sublayer includes a multi-head attention layer, a fully connected layer and two residual connection normalization layers.

4. The language steganalysis method based on knowledge injection and knowledge encoding according to claim 3 is characterized in that: The Mask-Transformer sublayer uses the visible matrix to control the correlation calculation between tokens during the attention calculation process; each hidden vector group (Q l ,K l ,V l ) is calculated as follows: Among them, W q 、W k and W v is a trainable parameter matrix, h l is the combination of all hidden vectors in layer l; S l is the attention score of layer l; if the vector h l i For h l j is visible, then the relevant calculation is the same as in BERT; if h l i and h l j is invisible, because M ij =-∞, attention score will be set to 0.

5. The language steganalysis method based on knowledge injection and knowledge encoding according to claim 1 is characterized in that: The process of GAT encoding knowledge of sparklines is as follows: First, the first layer uses a multi-head graph attention layer to extract the correlation information between nodes; for each multi-head graph attention layer, a globally shared weight matrix W is used to perform a linear transformation on the initial vector; Then, the Softmax method is used to calculate the graph attention coefficient α between each node and its connected nodes ij , the specific expression is as follows: Among them, N i Represents the node e i The set of all connected neighbor nodes, W∈R d×d' and a∈R 2d' is a model parameter, LeakyReLU is a nonlinear activation function; Finally, using α ij , the feature representation of node i is updated by adaptively aggregating information from neighboring nodes as follows: Where σ is the sigmoid activation function; the output vectors of multiple GALs are concatenated and fed into the next separate GAL to obtain the vector set e of m nodes 2 .

6. The language steganalysis method based on knowledge injection and knowledge encoding according to claim 1 is characterized in that: The cross entropy loss function is: Among them, y is the label of the normal vector, is the predicted label output by the model, and N is the number of input samples.

Images