Language steganalysis method based on knowledge injection and knowledge coding

Abstract

The invention discloses a language steganalysis method based on knowledge injection and knowledge coding, and belongs to the field of information hiding. Firstly, a language steganalysis system model based on knowledge injection and knowledge coding is constructed, the system model preprocesses input sentences with the help of a knowledge graph, and knowledge injection is completed. Then, BERT is adopted for the sentence tree and the visible matrix, extraction of semantic features is completed, and knowledge injection semantic coding is carried out. During preprocessing, a KE module for graph structure modeling is used for carrying out knowledge extraction on an input sentence and a knowledge graph to obtain a mini graph, and knowledge coding is carried out on the mini graph through GAT. Finally, the captured entity node features and the representation vectors obtained from the knowledge injection module are connected in series and added into a full connection layer, and the system model predicts whether an input text is a normal carrier or a steganography carrier through output of the full connection layer. According to the method, the problem of fragmentation of web texts is solved, and the result of language steganalysis is more accurate.

Description

Technical Field

[0001] The invention belongs to the field of information hiding, relates to text steganalysis detection, and specifically relates to a language steganalysis method based on knowledge injection and knowledge encoding. Background Art

[0002] Text steganalysis has developed rapidly with the advancement of natural language processing technology. Although detection capabilities have been significantly improved, it is still difficult to achieve satisfactory results when faced with real online text detection scenarios. One significant reason is that texts in public cyberspace, such as Twitter, are often highly fragmented, and individual texts usually have limited indicative features. Current detection models face challenges in extracting sufficiently rich features from individual input texts, hindering a comprehensive understanding of the text and limiting the model's ultimate detection capabilities.

[0003] Yang et al. [Sesy: Linguistic steganalysis framework integrating semantic and syntactic features] proposed a Sesy scheme, which uses a powerful pre-trained language model, Bidirectional Encoder Representation Transform (BERT), to extract semantic features, and designed a SeSy framework to consider the grammatical changes caused by embedding secret information, effectively improving the existing advanced language steganalysis algorithms. This scheme has enhanced the ability of text steganalysis to a certain extent, but these detection models face severe challenges caused by the fragmented nature of network text in real network detection scenarios.

[0004] Fu et al. [Hga:hierarchical feature extraction with graph and attention mechanism for linguistic steganalysis] proposed an Hga scheme, introducing LS-BGAT, an innovative method that merges text nodes with word nodes to form a large heterogeneous graph, and uses a graph attention network (GAT) to fully understand the relationship between words and corpora. However, this scheme still fails to effectively address the challenges of text fragmentation faced in online steganalysis. It is almost impossible to achieve a deep understanding of text based on a single sentence input. Summary of the invention

[0005] The present invention proposes a language steganalysis method based on knowledge injection and knowledge encoding to solve the practical challenges of text steganalysis caused by network text fragmentation.

[0006] The language steganalysis method based on knowledge injection and knowledge encoding has the following specific steps:

[0007] Step 1: Build a language steganalysis system model based on knowledge injection and knowledge encoding;

[0008] The system model is named KIKE, which includes two parts: semantic encoding based on knowledge injection and knowledge encoding based on knowledge extraction.

[0009] Step 2: The language steganalysis system model pre-processes the input sentence with the help of the knowledge graph to adapt to the existing sequential language model framework and complete the knowledge injection;

[0010] The specific preprocessing process is:

[0011] Step 201: The input sentence S is a set of tokens, represented by S = {w1, w2, ..., w n}, where n is the length of the sentence, w i is the i-th token of the sentence. The knowledge graph K is a graph structure consisting of triplets of entities and relationships between entities, denoted as K(V,E). K contains multiple triplets (v i ,e k ,v j ), where v i and v j is an entity, e k Represents relationships between entities.

[0012] Step 202, using the knowledge graph, converts the input sentence from its original form S to a sentence tree S':

[0013]

[0014] Among them, (w i ,...,w i+l-1 ) represents an entity E consisting of l words i , Indicates connection to E i Multiple branches.

[0015] Step 203, renumber the tokens in the sentence tree S', and use a combination of soft position index and hard position index to perform position embedding of the language model;

[0016] The specific implementation process is:

[0017] First, the soft position index of each token in the original sentence is 1 to n. If the distance between the content injected into the knowledge graph and the soft position i of the original sentence is x, the soft position index of the content injected into the knowledge graph is i+x, and is mapped to the same position vector in the position embedding process;

[0018] The hard position index is then used to reorder the tokens in the sentence tree and flatten the sentence tree into a linear sequence for the encoding stage.

[0019] Finally, the index of each token in the sentence tree is obtained by combining the above soft and hard position indexes.

[0020] Step 204, constructing a visible matrix M using hard position indexes to guide the visible range of the knowledge branch;

[0021] It can be seen that the matrix M is expressed as follows:

[0022]

[0023] in, Indicates that the two tags are on the same branch. Indicates that the two markers are not in the same branch. i and j are hard position indices.

[0024] Step 3: Use BERT to extract semantic features from the sentence tree and visibility matrix, and perform knowledge injection semantic encoding.

[0025] The specific process is:

[0026] Step 301, the embedding layer of BERT includes word embedding, position embedding and segment embedding. The soft position index is used in the position embedding part to represent the relative position of the token in the sentence, and the word embedding and segment embedding parts retain the original mapping method.

[0027] Step 302: embed these three parts of information into a high-dimensional feature space and add them together to obtain a hidden layer variable H0 = {h1, h2, ..., h n ′}∈R n′×d , where n' is the length of the flattened sequence after compressing the sentence tree, and d is the number of features.

[0028] Step 303, pass H0 to the Mask-Transformer module, which consists of L stacked Mask-Transformer sub-layers, and the input vector set of each layer is the output vector set of the previous layer, as shown below:

[0029] H l =Mask-Transformer l (H l-1 ),1≤l≤L

[0030] Each Mask-Transformer sublayer consists of a multi-head attention layer, a fully connected layer, and two residual connection normalization layers. The visible matrix is ​​used to control the correlation calculation between tokens during the attention calculation process. Each hidden vector group (Q l,K l ,V l ) is calculated as follows:

[0031]

[0032] Among them, W q , W k and W v is a trainable parameter matrix, h l is the combination of all hidden vectors in layer l. If the vector h l i For h l j is visible, then the relevant calculation is the same as in BERT. If h l i and h l j is invisible, because M ij =-∞, attention score Will be set to 0. S l is the attention score of the lth layer.

[0033] Step 304, after refining the L-layer semantic features, the representation vector of the knowledge injection semantic coding is obtained using the sigmoid function:

[0034]

[0035] Represents the hidden layer variable output by the L-th Mask-Transformer sublayer;

[0036] Step 4: During preprocessing, the KE module of graph structure modeling is used to extract knowledge from the input sentence and knowledge graph to obtain a mini-graph;

[0037] First, we extract the recognized entities from the input sentence and connect them one by one, and then connect them to the corresponding object branches in the knowledge graph to form a mini-graph G(γ,E), where γ = {v1,…,v m}, m is the number of entities in the sentence tree, E = {ε ab =(v a ,v b )1≤a,b≤m} is an edge set.

[0038] H is transformed into C The entity vector in is used as the initial vector of the graph node, and an adjacency matrix A∈R is used m×m To represent the connection relationship between entities.

[0039] Step 5: Encode the knowledge of the mini-graph through a two-layer GAT.

[0040] The specific implementation process is:

[0041] First, the first layer uses a multi-head graph attention layer to extract the correlation information between nodes. For each multi-head graph attention layer, a globally shared weight matrix W is used to linearly transform the initial vector.

[0042] Then, the Softmax method is used to calculate the graph attention coefficient α between each node and its connected nodes ij , the specific expression is as follows:

[0043]

[0044] Among them, N i Represents the node e i The set of all connected neighbor nodes, W∈R d×d' and is a trainable model parameter, and LeakyReLU is a nonlinear activation function.

[0045] Finally, using α ij , the feature representation of node i is updated by adaptively aggregating information from neighboring nodes as follows:

[0046]

[0047] Where σ is the sigmoid activation function. The output vectors of multiple GALs are concatenated and fed into the next separate GAL to obtain a vector set e of m nodes. 2 .

[0048] Step 6. After knowledge encoding is completed, the captured entity node features and the representation vector obtained from the knowledge injection module are concatenated and added to a fully connected layer. The system model predicts whether the input text is a normal carrier or a steganographic carrier through the output of the fully connected layer.

[0049] Step seven, train the prediction accuracy of the system model. During the training phase, use the cross entropy loss function to update the system model parameters. When the loss function is minimized, the training is completed.

[0050] The cross entropy loss function is:

[0051]

[0052] Among them, y is the label of the normal vector, is the predicted label output by the model, and N is the number of input samples.

[0053] The advantages of the present invention are:

[0054] 1. This invention proposes a language steganalysis method based on knowledge injection and knowledge encoding to solve the fragmentation problem of network text in the real online environment. It can be done through the semantic extraction part, cognitive verification part and information fusion part of knowledge injection. By introducing professional domain knowledge with the help of knowledge graph, the language model can better assist in performing classification tasks.

[0055] 2. In order to realize a new type of language steganalysis based on knowledge injection and knowledge encoding, the present invention injects triples into sentences with the help of knowledge graphs from professional fields in knowledge injection, and extracts semantic features together with the generated sentence tree and visible matrix and the existing Transformer architecture. In knowledge encoding, entities are integrated into mini-graphs and cognitive verification is performed using graph neural networks. The combination of the above two processes makes the results of language steganalysis more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a language steganalysis system model based on knowledge injection and knowledge encoding proposed by the present invention;

[0057] Figure 2 This is the effect of inserting different numbers of branches on the detection effect F1 value in the embodiment. DETAILED DESCRIPTION

[0058] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0059] Inspired by the innovative application of a new language steganalysis framework with external knowledge, this paper mainly focuses on designing a more powerful text feature extraction model using most text steganalysis algorithms in order to achieve efficient steganalysis. However, in real online environments, they face significant challenges related to the fragmentation of network texts, which greatly limits their practical applicability.

[0060] This paper proposes a language steganalysis method based on knowledge injection and knowledge encoding, which is named KIKE. The core idea of ​​the KIKE model is to introduce external knowledge and guide the fine-tuning process of entity vectors from both the data layer and the model layer, which is equivalent to adding annotations to entities in the text to help the model better understand the text.

[0061] In knowledge injection, triples are injected into sentences with the help of knowledge graphs from professional fields, and semantic features are extracted by combining the generated sentence tree and visibility matrix with the existing Transformer architecture. In knowledge encoding, entities are integrated into mini-graphs and cognitive verification is performed using graph neural networks. The protocol flow of each stage is as follows: Figure 1The proposed algorithm can be divided into two types: semantic coding algorithm based on knowledge injection and knowledge coding algorithm based on knowledge extraction. Knowledge extraction occurs in the preprocessing stage, and the semantic coding part integrates external information.

[0062] The specific steps are as follows:

[0063] Step 1: Build a language steganalysis system model based on knowledge injection and knowledge encoding;

[0064] The system model includes the semantic extraction part of knowledge injection, the cognitive verification part and the information fusion part. By introducing professional domain knowledge with the help of knowledge graph, the language model can better assist in performing classification tasks. Figure 1 As shown, the proposed system model is divided into two parts: semantic encoding based on knowledge injection at the bottom and knowledge encoding based on knowledge extraction at the top.

[0065] Semantic encoding based on knowledge injection: Triples are injected into sentences with the help of knowledge graphs from professional fields, and the generated sentence trees and visibility matrices are combined with the existing Transformer architecture to extract semantic features.

[0066] Knowledge encoding based on knowledge extraction: responsible for integrating entities into mini-graphs and using graph neural networks for cognitive verification.

[0067] Step 2: In order to better integrate knowledge into sentences and adapt to the existing sequential language model framework, the system model preprocesses the input sentences with the help of knowledge graphs before extracting semantic features, which is called "knowledge injection".

[0068] The specific preprocessing process is:

[0069] Step 201: represent the input sentence as S = {w1, w2, ..., w n}, where n is the length of the sentence, w i is the i-th token of the sentence. The knowledge graph is represented as K(V,E), where K contains multiple triples (v i ,e k ,v j ), where v i and v j is an entity, e k Represents relationships between entities.

[0070] In this step, the input sentence and knowledge graph are represented in the form of data structures, where the sentence S is a set of tokens, each token represents a word or logo in the sentence. The knowledge graph K is a graph structure consisting of triplets of entities and relationships between entities. Such a representation facilitates subsequent processing and analysis.

[0071] Step 202, using the knowledge graph, converts the input sentence from its original form S to a sentence tree S':

[0072]

[0073] Among them, (w i ,...,w i+l-1 ) represents an entity E consisting of l words i , Indicates connection to E i Multiple branches.

[0074] In this step, the original sentence S is transformed into a more structured form, where the entity E i It contains a series of related words and is connected to other entities through branches to form a sentence tree structure. Such a transformation helps capture the relevance and structural information between words in a sentence.

[0075] Step 203: Since the linear sequence is converted into a tree structure, the tokens in S' need to be renumbered to adapt to the position embedding and attention calculation mechanism of the advanced language model. The embedding is performed using a combination of soft position index and hard position index, and the soft position index is used to ensure that the context structure information is not affected during the position embedding process;

[0076] Figure 1 In the embodiment shown, the input sentence is "We use syntactic tree pruning strategy to optimize semantic space", and the specific implementation process of soft position index and hard position index in the position embedding process is:

[0077] Calculation of soft position index: For each token in the sentence tree, the soft position index is calculated. For example, for "graph" and "information", their distance from "pruning" in the sentence tree is 2, so their soft position index is 7. These soft position indexes will be used to map to the corresponding position vectors in the position embedding process.

[0078] Calculation of hard position index: The hard position index is used to reorder the tokens in the sentence tree and flatten the sentence tree into a linear sequence for subsequent encoding stages. For example, with the hard position index, the index of "graph" is 7, while the index of "information" is 10.

[0079] The combination of soft and hard position indexes ensures that the structural information of the sentence tree is taken into account during the conversion process, while also adapting to the language model's needs for position embedding and attention calculation. This helps preserve contextual information and improves the model's ability to understand and process input data.

[0080] Step 204, since the knowledge injection method introduces external tags into the representation learning process, it is necessary to ensure that the knowledge branches are only used to enrich the meaning of their corresponding entities and avoid interfering with the semantic representation of other words in the original sentence. Therefore, the visible matrix M is constructed using hard position indexes as shown below:

[0081]

[0082] in, indicates that the two tags are on the same branch, and Indicates that they are not in the same branch. i and j are hard position indices.

[0083] The visibility matrix M constructed in this way guides the visible range of the knowledge branch through hard position indexing, ensuring that the injected knowledge only affects the corresponding entity without disturbing the semantic representation of other words in the original sentence. This is an important step to maintain the robustness and understanding of the model.

[0084] Step 3: In order to explore the rich meanings of words from multiple perspectives and capture the long-distance dependencies between characters, BERT is used in practice to complete the task of semantic feature extraction.

[0085] The embedding layer of BERT consists of three parts: word embedding, position embedding, and segment embedding. The embedding process is:

[0086] First, soft position index is used in the position embedding part to represent the relative position of tokens in the sentence. The other two parts retain the original mapping method.

[0087] Then, after embedding these three types of information into the high-dimensional feature space, they are added together to obtain the hidden layer variables H0 = {h1,h2,...,h n′}∈R n′×d , where n' is the length of the flattened sequence after compressing the sentence tree, and d is the number of features.

[0088] Finally, H0 is passed to the Mask-Transformer module, which consists of L stacked Mask-Transformer sublayers. The input vector set of each layer is the output vector set of the previous layer, as shown below:

[0089] H l =Mask-Transformer l (H l-1 ),1≤l≤L

[0090] The calculation of each layer includes a multi-head attention layer, a fully connected layer, and two residual connection normalization layers. Unlike the BERT model, this paper uses a visible matrix to control the correlation calculation between tokens during the attention calculation process. Formally, the calculation of each hidden vector group is as follows:

[0091]

[0092] Among them, W q , W k and W v is a trainable parameter matrix, h l is the combination of all hidden vectors in the lth layer. As shown in the formula, if the vector h l i For h l j is visible, then the relevant calculation is the same as in BERT. If h l i and h l j is invisible, because M ij =-∞, attention score will be set to 0.

[0093] After L-layer semantic feature refinement, the representation vector of knowledge injection semantic encoding is obtained by using the Sigmoid function:

[0094]

[0095] Word embedding: The word embedding part of the BERT model is used, retaining the original mapping method.

[0096] Position Embedding: In the position embedding part, soft position index is used to represent the relative position of tokens in the sentence to preserve the order information between tokens.

[0097] Segment Embedding: Keep the original segment embedding mapping method.

[0098] Mask-Transformer module: H0 is passed to the Mask-Transformer module, which consists of L stacked Mask-Transformer sublayers. The input vector set of each layer is the output vector set of the previous layer. The calculation of each layer includes a multi-head attention layer, a fully connected layer, and two residual connection normalization layers. Among them, the visible matrix is ​​used to control the correlation calculation between tokens during the attention calculation process.

[0099] Sigmoid function: After L layers of semantic feature refinement, the representation vector of knowledge-injected semantic coding is obtained by using the Sigmoid function.

[0100] This process aims to capture the rich meaning of words from multiple perspectives while considering the long-distance dependencies between characters. A visible matrix is ​​introduced through the Mask-Transformer module to control the relevance of attention calculation.

[0101] Step 4, the semantic encoding part integrates external information and can be independently input into the discriminator to complete the detection task. However, in order to better explore the potential relationship between entity concepts from a cognitive perspective and identify the knowledge consistency in the content from a cognitive perspective, the present invention designs a KE module using graph structure modeling, which includes two parts: knowledge extraction and knowledge encoding.

[0102] Knowledge extraction occurs in the preprocessing stage, extracting the recognized entities from the sentence and connecting them one by one, and then connecting them to the corresponding object branches in the knowledge graph to form a small graph G(γ,E), where γ = {v1,…,v m}, m is the number of entities in the sentence tree, E = {ε ab =(v a ,v b )1≤a,b≤m} is an edge set.

[0103] H is transformed into C The entity vector in is used as the initial vector of the graph node, and an adjacency matrix A∈R is used m×m To represent the connection relationship between entities.

[0104] This part aims to use graph structures to model the relationship between entity concepts, and to better identify knowledge consistency in content through small graphs formed by knowledge extraction. The process of knowledge encoding is carried out through graph neural networks (GNNs) to further capture the semantic relationship between entities. This helps to better understand the entity associations in the text from a cognitive perspective.

[0105] In step 5, a two-layer GAT (Graph Attention Network) is applied to encode the mini-graph. It can simultaneously extract the local relationship of text nodes and the global structure of the entire graph, as well as infer similar structures and features from the learned graph data, with good generalization performance.

[0106] The specific implementation process is:

[0107] First, the first layer uses a multi-head graph attention layer to explore the association information between nodes from different perspectives. For each multi-head graph attention layer, a globally shared weight matrix W is used to linearly transform the initial vector.

[0108] Then, the Softmax method is used to calculate the graph attention coefficient α between each node and its connected nodes ij , the specific expression is as follows:

[0109]

[0110] Among them, N i Represents the node e i The set of all connected neighbor nodes, W∈R d×d' and a∈R 2d' is a trainable model parameter, and LeakyReLU is a nonlinear activation function.

[0111] Using alpha ij , the feature representation of node i can be updated by adaptively aggregating information from neighboring nodes as follows:

[0112]

[0113] Where σ is the sigmoid activation function. The output vectors of multiple GALs are concatenated and then fed into the next separate GAL (Graph Attentional Layer) to obtain the vector set e of m nodes. 2 .

[0114] The purpose of this whole step is to capture the complex relationship between text nodes through the multi-head graph attention mechanism, making full use of the information of the graph structure to obtain a higher-level semantic representation. This helps the model better understand the association between entities and improves the model's ability to understand and express text.

[0115] Step 6, once the knowledge encoding is completed, the captured entity node features and the representation vector obtained from the knowledge injection module are concatenated and added to a fully connected layer to determine whether the text is a stego carrier.

[0116] During the training phase, the model parameters are updated using the following cross entropy loss function:

[0117]

[0118] Among them, y is the label of the normal vector, is the predicted label output by the model, and N is the number of input samples.

[0119] Information Fusion: Once knowledge encoding is completed, i.e., entity node features and representation vectors obtained from the knowledge injection module, these two aspects of information are concatenated. This is to make full use of different levels and types of information obtained through the knowledge injection and knowledge encoding stages to improve the model's overall understanding of the text.

[0120] Fully connected layer: The concatenated feature vectors are input into a fully connected layer. The fully connected layer is a common neural network layer used to learn complex relationships between features. Here, its role is to further combine and transform the input features to prepare for the final classification task.

[0121] Text detection task: The final fully connected layer output is used to determine whether the text is a stego carrier. This is achieved through the binary classification output of the model, that is, the model predicts whether the input text is a normal carrier or a stego carrier. The purpose of this task is to make the model have good text detection performance through parameter updates during the training process.

[0122] Loss function: During the training phase, the cross entropy loss function is used as the optimization target of the model. The cross entropy loss function is used to measure the difference between the probability distribution of the model output and the actual label. By minimizing this difference, the model parameters are adjusted to make the model more accurately predict the category of the text.

[0123] The purpose of the whole step is to combine the information obtained through knowledge injection and knowledge encoding into the final text detection task, thereby improving the model's detection performance for network text steganography.

[0124] The present invention analyzes the effectiveness of the proposed scheme and verifies the effectiveness of the proposed language steganalysis scheme based on knowledge injection and knowledge encoding by providing performance analysis.

[0125] Verification 1: The proposed scheme and baseline model were tested for the detection performance of text steganography algorithms of various data sets at different embedding rates, and the results are shown in Table 1. In general, the scheme proposed in the present invention is better than all other baseline models. First, thanks to the powerful semantic understanding ability of the BERT model, the detection model based on BERT is significantly better than other deep learning methods, and can improve the accuracy Acc by about 8%. When the FLC embedding amount is 3.98bpw, KIKE's Acc reaches 98.6%. Secondly, the BERT model that is only applicable in this scenario is usually not so stable in the balance between precision P and recall rate R, and the use of the Knowledge Infusion strategy can achieve better and more robust results.

[0126] Verification 2: Ablation experiments were conducted in Table 2 to quantify the impact of the key components of the proposed model. First, without changing the original structure and parameter size of the BERT model, adding the KI module alone can effectively improve the detection accuracy by about 2% to 3%, proving the effectiveness of the data enhancement strategy of integrating knowledge into sentences. Secondly, by comparing the experimental results of KI and KIKE, it can be found that when bpw is low, the lack of entity relevance in the sentence makes the language model based on data enhancement play a greater role. However, when bpw gradually increases, the added KE module can better detect stego traces from a cognitive perspective by checking the consistency of entity semantics.

[0127] Verification 3: A deeper exploration of the number of branches of introduced knowledge was conducted, such as Figure 2 As shown. Overall, first of all, both KI and KIKE work best when the number of branches is 3. This is because appropriate supplementary knowledge helps to fully characterize the semantics of professional entities, rather than relying solely on the role of each independent word in the entity in the language model to affect the meaning of the entire sentence. However, when the number of branches rises to 4, the detection effect will decay. It is speculated that too much additional information will cause words to deviate from the original word vector space position and disrupt the semantic expression of the original sentence.

[0128] During the experiment, the weight of the Knowlede Encoding input into the classifier was controlled according to the adjustment of different parameters of GAT. It was observed that a low number of branches is suitable for knowledge encoding with a small number of parameters, because a knowledge graph with too few branches requires high accuracy of the introduced knowledge, and one-sided knowledge may affect the accuracy of the language model; while a high number of branches is suitable for knowledge encoding with a larger number of parameters, which can make the knowledge graph play a role, but it should not be too many, because it still needs to be judged based on the word logic of the sentence itself.

[0129] Table 1 Detection performance of each model for different text steganography algorithms at different embedding rates

[0130]

[0131] Table 2 Comparison of text steganalysis performance of each module in KIKE

[0132]

Claims

1. A language steganalysis method based on knowledge injection and knowledge encoding, characterized in that: The specific steps are as follows: Step 1: Build a language steganalysis system model based on knowledge injection and knowledge encoding; The system model is named KIKE, which includes semantic coding based on knowledge injection and knowledge coding based on knowledge extraction; Step 2: The language steganalysis system model pre-processes the input sentence with the help of the knowledge graph to adapt to the existing sequential language model framework and complete the knowledge injection; The specific preprocessing process is: Step 201: The input sentence S is a set of tokens, represented by S = {w1, w2, ..., w n }, where n is the length of the sentence, w i is the i-th token of the sentence; the knowledge graph K is a graph structure consisting of triplets of entities and relationships between entities, denoted as K(V,E), where K contains multiple triplets (v i ,e k ,v j ), where v i and v j is an entity, e k Represents relationships between entities; Step 202, using the knowledge graph, converts the input sentence from its original form S to a sentence tree S': Among them, (w i ,...,w i+l-1 ) represents an entity E consisting of l words i , Indicates connection to E i Multiple branches of Step 203, renumber the tokens in the sentence tree S', and use a combination of soft position index and hard position index to perform position embedding of the language model; Step 204, constructing a visible matrix M using hard position indexes to guide the visible range of the knowledge branch; It can be seen that the matrix M is expressed as follows: in, Indicates that the two tags are on the same branch. Indicates that the two tags are not in the same branch, i and j are hard position indexes; Step 3: Use BERT to extract semantic features from the sentence tree and visibility matrix, and perform knowledge injection semantic encoding. The specific process is: Step 301, the embedding layer of BERT includes word embedding, position embedding and segment embedding. The soft position index is used in the position embedding part to represent the relative position of the token in the sentence. The word embedding and segment embedding parts retain the original mapping method. Step 302: embed these three parts of information into a high-dimensional feature space and add them together to obtain a hidden layer variable H0 = {h1, h2, ..., h n′ }∈R n′×d , where n' is the length of the flattened sequence after compressing the sentence tree, and d is the number of features; Step 303, pass H0 to the Mask-Transformer module, which consists of L stacked Mask-Transformer sub-layers, and the input vector set of each layer is the output vector set of the previous layer, as shown below: H l =Mask-Transformer l (H l-1 ),1≤l≤L Step 304, after refining the L-layer semantic features, the representation vector of the knowledge injection semantic coding is obtained using the sigmoid function: Among them, H0 L Represents the hidden layer variable output by the L-th Mask-Transformer sublayer; Step 4: During preprocessing, the KE module of graph structure modeling is used to extract knowledge from the input sentence and knowledge graph to obtain a mini-graph; First, we extract the recognized entities from the input sentence and connect them one by one, and then connect them to the corresponding object branches in the knowledge graph to form a mini-graph G(γ,E), where γ = {v1,…,v m }, m is the number of entities in the sentence tree, E = {ε ab =(v a ,v b )1≤a,b≤m} is an edge set; H is transformed into C The entity vector in is used as the initial vector of the graph node, and an adjacency matrix A∈R is used m×m To represent the connection relationship between entities. Step 5: Knowledge encoding of the mini-graph is performed through a two-layer GAT. Then the captured entity node features and the representation vector obtained from the knowledge injection module are concatenated and added to a fully connected layer. The system model predicts whether the input text is a normal carrier or a steganographic carrier through the output of the fully connected layer. Step six, train the prediction accuracy of the system model. During the training phase, use the cross entropy loss function to update the system model parameters. When the loss function is minimized, the training is completed.

2. According to the method of language steganalysis based on knowledge injection and knowledge encoding according to claim 1, it is characterized in that: The specific implementation process of renumbering the tags in the sentence tree S' is: First, the soft position index of each token in the original sentence is 1 to n. If the distance between the content injected into the knowledge graph and the soft position i of the original sentence is x, the soft position index of the content injected into the knowledge graph is i+x, and is mapped to the same position vector in the position embedding process; The hard position index is then used to reorder tokens in the sentence tree and flatten the sentence tree into a linear sequence for the encoding stage; Finally, the index of each token in the sentence tree is obtained by combining the above soft and hard position indexes.

3. According to the method of language steganalysis based on knowledge injection and knowledge encoding according to claim 1, it is characterized in that: Each Mask-Transformer sublayer includes a multi-head attention layer, a fully connected layer and two residual connection normalization layers.

4. According to claim 3, a language steganalysis method based on knowledge injection and knowledge encoding is characterized in that: The Mask-Transformer sublayer uses the visible matrix to control the correlation calculation between tokens during the attention calculation process; each hidden vector group (Q l ,K l ,V l ) is calculated as follows: Among them, W q , W k and W v is a trainable parameter matrix, h l is the combination of all hidden vectors in the lth layer; S l is the attention score of the lth layer; if the vector h l i For h l j is visible, then the relevant calculation is the same as in BERT; if h l i and h l j is invisible, because M ij =-∞, attention score will be set to 0.

5. According to the method of language steganalysis based on knowledge injection and knowledge encoding according to claim 1, it is characterized in that: The process of GAT encoding knowledge of sparklines is as follows: First, the first layer uses a multi-head graph attention layer to extract the correlation information between nodes; for each multi-head graph attention layer, a globally shared weight matrix W is used to linearly transform the initial vector; Then, the Softmax method is used to calculate the graph attention coefficient α between each node and its connected nodes ij , the specific expression is as follows: Among them, N i Represents the node e i The set of all connected neighbor nodes, W∈R d×d' and a∈R 2d' is a model parameter, LeakyReLU is a nonlinear activation function; Finally, using α ij , the feature representation of node i is updated by adaptively aggregating information from neighboring nodes as follows: Where σ is the sigmoid activation function; the output vectors of multiple GALs are concatenated and fed into the next separate GAL to obtain the vector set e of m nodes 2 .

6. The language steganalysis method based on knowledge injection and knowledge encoding according to claim 1 is characterized in that: The cross entropy loss function is: Among them, y is the label of the normal vector, is the predicted label output by the model, and N is the number of input samples.

Images