Tika-based file confidentiality detection implementation method

Abstract

A file confidentiality detection implementation method based on Tika comprises the steps that a sensitive information word bank is obtained, and sensitive vocabularies in a target text or a target code are detected and locked based on the sensitive information word bank. And according to the context information of the sensitive vocabularies, constructing a logic binomial tree corresponding to the sensitive vocabularies through semantic analysis. A word embedding vector corresponding to each node in the logic binomial tree is calculated through a word embedding model, the similarity between any two nodes is calculated based on the word embedding vectors, and different nodes are classified. And taking the word embedding vector as an input of a deep learning network model, and calling a deep learning network to detect the word embedding vector to obtain a confidentiality detection result of the target text or the target code. The relationship between father nodes and child nodes in the logic binomial tree is used for representing the causal relationship of the context information of the sensitive vocabularies, and the relationship between brother nodes is used for representing the logic nesting relationship of the context information of the sensitive vocabularies.

Description

Technical Field

[0001] The present invention belongs to the technical field of text data security, and more specifically, relates to a method for realizing file confidentiality detection based on Tika. Background Art

[0002] With the acceleration of the digitalization process, data has become one of the most important assets of enterprises and organizations. However, data security risks are also increasing, especially the increasingly diversified channels of penetration through the supply chain and industrial chain. The state has successively issued relevant laws and regulations on data security, explicitly requiring the strengthening of data security protection and the prevention of sensitive information leakage. In this context, the confidentiality detection of files is particularly important. As an information carrier, files are widely used in various business activities. Ensuring the security of confidential information in files is not only a necessary measure to comply with laws and regulations, but also an important means to protect business secrets, maintain customer privacy, and prevent internal information leakage. Therefore, the development of efficient and accurate file confidentiality detection methods has become a key demand in the field of data security.

[0003] Apache Tika is an open source framework developed by the Apache Software Foundation, which is designed to detect and extract metadata and content from various document types. It provides a unified interface and supports parsing of multiple file formats, including text, images, audio, video, documents, and compressed files. Tika has a wide range of applications in content extraction and metadata parsing, especially when processing common document formats such as Word and PDF.

[0004] At present, the file detection method based on Tika mainly includes the following steps: content extraction: using Tika's parser to extract the text content and metadata information of the file; content analysis: keyword matching, regular expression detection, or combining machine learning, deep learning (such as natural language processing) and other technologies to identify sensitive information; result output: generating a detection report or warning information based on the analysis results. This method can effectively detect obvious sensitive information when processing ordinary text files and documents, and has certain practicality.

[0005] However, the existing Tika-based file confidentiality detection methods have obvious deficiencies when processing code files (such as .ipynb, .py, .c, .cpp, etc.): Limited parsing ability: Tika's parsing of code files mainly stays at the plain text level, and it is impossible to deeply understand the grammatical structure and logical relationship of the code. Code files have special programming language syntax and contain complex structures such as variables, functions, classes, and comments. It is difficult to obtain meaningful content by directly extracting text. Sensitive information is highly hidden: Sensitive information in the code (such as hard-coded passwords, keys, API keys, personal identity information, etc.) is usually hidden in variable assignments, configuration files, or comments, and may be encoded or obfuscated. Traditional keyword matching and regular expression methods are difficult to fully cover and are prone to omissions or false positives. Multi-language and format complexity: Code files involve multiple programming languages, and the syntax and structure of different languages ​​vary greatly. For example, Python code (such as .py or .ipynb files) and C / C++ code (such as .c, .cpp files) have significant differences in syntax and file structure. Tika lacks professional parsers for different programming languages ​​and cannot effectively process diverse code files. Poor applicability of existing models: Existing machine learning and deep learning models are mostly trained on natural language texts, and lack the ability to understand and analyze code languages, making it difficult to effectively identify sensitive information in the code. Therefore, existing technologies have significant defects in confidentiality detection of code files and cannot meet actual application needs. Summary of the invention

[0006] In order to solve the deficiencies in the prior art, the purpose of the present invention is to solve the above-mentioned defects and further propose a file confidentiality detection implementation method based on Tika.

[0007] The present invention adopts the following technical solution.

[0008] The first aspect of the present invention discloses a method for realizing file confidentiality detection based on Tika, the method comprising:

[0009] Acquire a sensitive information word library, and detect and lock sensitive words in the target text or target code based on the sensitive information word library;

[0010] According to the context information of the sensitive words, construct a logical binomial tree corresponding to the sensitive words through semantic analysis;

[0011] Calculate the word embedding vector corresponding to each node in the logical binomial tree through the word embedding model, and calculate the similarity between any two nodes based on the word embedding vector, wherein the similarity is used to classify different nodes;

[0012] Using the word embedding vector as an input of a deep learning network model to call the deep learning network to detect the word embedding vector and obtain a confidentiality detection result of the target text or target code;

[0013] The relationship between the parent node and the child node in the logical binomial tree is used to represent the causal relationship of the context information of the sensitive words, and the relationship between the sibling nodes is used to represent the logical nested relationship of the context information of the sensitive words.

[0014] Furthermore, the step of acquiring a sensitive information word library and detecting and locking sensitive words in a target text or target code based on the sensitive information word library includes:

[0015] Parse all subfiles in the file directory to be tested for confidentiality by Parser, and convert the parsing results into strings, wherein the parsing results include sensitive words in the file to be tested for confidentiality;

[0016] The sensitive words are extracted from the analysis results based on the pre-input sensitive information word library, and the sensitive words are locked.

[0017] Furthermore, the word embedding model is used to calculate the word embedding vector corresponding to each node in the logical binomial tree, and the similarity between any two nodes is calculated based on the word embedding vector, and the similarity is used to classify different nodes, including:

[0018] The word embedding vector corresponding to each node in the logical binomial tree is obtained through the Word2Vec word embedding model, and the word embedding vector corresponding to the node v in the logical binomial tree is obtained based on the word embedding vector. k The associated adjacency matrix V k ;

[0019] Wherein, the expression of the adjacency matrix is:

[0020]

[0021] In the formula, if i=j, v ij Represents the i-th child node of a vertex in a logical binomial tree. If i≠j, v ij Indicates that in the sub-binomial tree with the i-th child node as the root node, the j-th leaf node to the node v k The distance between them, i,j = 1, 2,…, H, where H represents the depth of the logical binomial tree.

[0022] Furthermore, the word embedding model is used to calculate the word embedding vector corresponding to each node in the logical binomial tree, and the similarity between any two nodes is calculated based on the word embedding vector, and the similarity is used to classify different nodes, and then it also includes:

[0023] According to the data types obtained by classifying different nodes according to similarity, and based on the values ​​corresponding to the nodes of the corresponding data types collected at any time, combined with the preset weight learning matrix, the mapping vector of each node in the logical binomial tree is calculated; and

[0024] Based on a node set of the same data type in the logical binomial tree and an adjacency matrix associated with nodes in the node set, a neighborhood feature vector of any node in the logical binomial tree with respect to any data type is calculated.

[0025] Furthermore, the word embedding model is used to calculate the word embedding vector corresponding to each node in the logical binomial tree, and the similarity between any two nodes is calculated based on the word embedding vector, and the similarity is used to classify different nodes, and then it also includes:

[0026] Call the activation function to calculate the attention scores of all data types based on the preset attention vector of any data type;

[0027] According to the attention scores of all data types in the logical binomial tree, the influence factor between any two nodes is calculated.

[0028] Furthermore, the word embedding model is used to calculate the word embedding vector corresponding to each node in the logical binomial tree, and the similarity between any two nodes is calculated based on the word embedding vector, and the similarity is used to classify different nodes, and then it also includes:

[0029] Based on a preset confidence threshold, child nodes of the same data type are merged, and according to the influence factor between any two nodes of the same data type, an activation function is called to calculate the feature vector of each data type; and

[0030] A logical binomial tree network model is established according to the heap embedding matrix of the logical binomial tree, and the weight matrix in the logical binomial tree network model is initialized, and the word embedding vector is converted into a query vector, a key vector and a value vector through a self-attention layer.

[0031] Further, the word embedding vector is used as an input of a deep learning network model to call the deep learning network to detect the word embedding vector to obtain a confidentiality detection result of the target text or target code, including:

[0032] A prediction model based on a cross entropy loss function, and updating the weights and biases of the logistic binomial tree network model according to a feedforward network;

[0033] Predicting the current file to be detected according to the trained logical binomial tree network model, and obtaining the confidentiality detection result of the current file to be detected based on the word embedding vector of the current file to be detected;

[0034] Among them, the feedforward network is represented by two layers of linear transformation, namely the first layer of linear transformation and the second layer of linear transformation. The feedforward network is obtained by calling the activation function to process the weights and biases of the first layer of linear transformation and the second layer of linear transformation.

[0035] The second aspect of the present invention discloses a device for realizing file confidentiality detection based on Tika, the device comprising:

[0036] A sensitive word locking module, used to obtain a sensitive information word library, and detect and lock sensitive words in the target text or target code based on the sensitive information word library;

[0037] A logical binomial tree construction module, used to construct a logical binomial tree corresponding to the sensitive words through semantic analysis according to the context information of the sensitive words;

[0038] A node classification module, used to calculate the word embedding vector corresponding to each node in the logical binomial tree through a word embedding model, and calculate the similarity between any two nodes based on the word embedding vector, wherein the similarity is used to classify different nodes;

[0039] A confidentiality detection module, used to use the word embedding vector as an input of a deep learning network model to call the deep learning network to detect the word embedding vector and obtain a confidentiality detection result of the target text or target code;

[0040] The relationship between the parent node and the child node in the logical binomial tree is used to represent the causal relationship of the context information of the sensitive words, and the relationship between the sibling nodes is used to represent the logical nested relationship of the context information of the sensitive words.

[0041] The third aspect of the present invention discloses a terminal, including a processor and a storage medium; the characteristics are:

[0042] The storage medium is used to store instructions;

[0043] The processor is used to operate according to the instructions to execute the steps of the method described in the first aspect.

[0044] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0045] The beneficial effects of the present invention are that, compared with the prior art, the present invention has the following advantages:

[0046] By obtaining a sensitive information word library, and based on the sensitive information word library, the sensitive words in the target text or target code are detected and locked. According to the context information of the sensitive words, a logical binomial tree corresponding to the sensitive words is constructed through semantic analysis. The word embedding vector corresponding to each node in the logical binomial tree is calculated through the word embedding model, and the similarity between any two nodes is calculated based on the word embedding vector to classify different nodes. The word embedding vector is used as the input of the deep learning network model to call the deep learning network to detect the word embedding vector and obtain the confidentiality detection result of the target text or target code. This method can not only train on natural language text, but also understand and analyze the code language, thereby effectively identifying sensitive information in the code, and has good applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of a method for realizing file confidentiality detection based on Tika provided by the present invention;

[0048] Figure 2A This is one of the schematic diagrams of the code confidentiality detection logic binomial tree architecture of a Tika-based file confidentiality detection implementation method provided by the present invention;

[0049] Figure 2B This is the second schematic diagram of the code confidentiality detection logic binomial tree architecture of a Tika-based file confidentiality detection implementation method provided by the present invention;

[0050] Figure 2C This is the third schematic diagram of the code confidentiality detection logic binomial tree architecture of a Tika-based file confidentiality detection implementation method provided by the present invention. DETAILED DESCRIPTION

[0051] The present application is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present application.

[0052] like Figure 1 As shown, in one embodiment, a method for implementing file confidentiality detection based on Tika includes the following steps:

[0053] Step S110, obtaining a sensitive information word library, and detecting and locking sensitive words in the target text or target code based on the sensitive information word library.

[0054] In some embodiments, the present invention provides a method for implementing file confidentiality detection based on Tika, which obtains a sensitive information word library, and detects and locks sensitive words in a target text or target code based on the sensitive information word library, specifically comprising the following steps:

[0055] Step S111, all sub-files in the file directory to be checked for confidentiality are parsed by Parser, and the parsing results are converted into character strings, wherein the parsing results include sensitive words in the file to be checked for confidentiality.

[0056] Step S112: extract sensitive words from the parsing result based on the pre-input sensitive information word library, and lock the sensitive words.

[0057] Step S120: construct a logical binomial tree corresponding to the sensitive words through semantic analysis according to the context information of the sensitive words.

[0058] The relationship between the parent node and the child node in the logical binomial tree is used to represent the causal relationship of the context information of the sensitive words, and the relationship between the sibling nodes is used to represent the logical nested relationship of the context information of the sensitive words.

[0059] Step S130, calculating the word embedding vector corresponding to each node in the logical binomial tree through the word embedding model, and calculating the similarity between any two nodes based on the word embedding vector, and the similarity is used to classify different nodes.

[0060] In some embodiments, the present invention provides a Tika-based file confidentiality detection implementation method, which calculates the word embedding vector corresponding to each node in the logical binomial tree through a word embedding model, and calculates the similarity between any two nodes based on the word embedding vector, and the similarity is used to classify different nodes, specifically including the following steps:

[0061] Step S131, obtain the word embedding vector corresponding to each node in the logical binomial tree through the Word2Vec word embedding model, and obtain the word embedding vector corresponding to the node v in the logical binomial tree based on the word embedding vector k The associated adjacency matrix V k .

[0062] The expression of the adjacency matrix is:

[0063]

[0064] In the formula, if i=j, v ij Represents the i-th child node of a vertex in a logical binomial tree. If i≠j, v ij Indicates that in the sub-binomial tree with the i-th child node as the root node, the j-th leaf node to the node v kThe distance between them, i,j = 1, 2,…, H, where H represents the depth of the logical binomial tree.

[0065] In some embodiments, the present invention provides a Tika-based file confidentiality detection implementation method, which calculates the word embedding vector corresponding to each node in the logical binomial tree through the word embedding model, and calculates the similarity between any two nodes based on the word embedding vector, and the similarity is used to classify different nodes, and then further includes the following steps:

[0066] Step S210, based on the data types obtained by classifying different nodes according to similarity, and based on the values ​​corresponding to the nodes of the corresponding data type collected at any time, combined with the preset weight learning matrix, calculate the mapping vector of each node in the logical binomial tree.

[0067] Step S220, based on the node set of the same data type in the logical binomial tree and the adjacency matrix associated with the nodes in the node set, calculate the neighborhood feature vector of any node in the logical binomial tree with respect to any data type.

[0068] In some embodiments, the present invention provides a Tika-based file confidentiality detection implementation method, which calculates the word embedding vector corresponding to each node in the logical binomial tree through the word embedding model, and calculates the similarity between any two nodes based on the word embedding vector, and the similarity is used to classify different nodes, and then further includes the following steps:

[0069] Step S230, calling the activation function to calculate the attention scores of all data types according to the preset attention vector of any data type.

[0070] Step S240, calculate the influence factor between any two nodes based on the attention scores of all data types in the logical binomial tree.

[0071] In some embodiments, the present invention provides a Tika-based file confidentiality detection implementation method, which calculates the word embedding vector corresponding to each node in the logical binomial tree through the word embedding model, and calculates the similarity between any two nodes based on the word embedding vector, and the similarity is used to classify different nodes, and then further includes the following steps:

[0072] Step S250, based on a preset confidence threshold, the child nodes of the same data type are merged, and according to the influence factor between any two nodes of the same data type, an activation function is called to calculate the feature vector of each data type.

[0073] Step S260, establish a logical binomial tree network model according to the heap embedding matrix of the logical binomial tree, initialize the weight matrix in the logical binomial tree network model, and convert the word embedding vector into a query vector, a key vector and a value vector through a self-attention layer.

[0074] Step S140, using the word embedding vector as the input of the deep learning network model to call the deep learning network to detect the word embedding vector and obtain the confidentiality detection result of the target text or target code.

[0075] In some embodiments, the present invention provides a method for implementing file confidentiality detection based on Tika, which uses a word embedding vector as the input of a deep learning network model to call the deep learning network to detect the word embedding vector and obtain a confidentiality detection result of the target text or target code, specifically comprising the following steps:

[0076] Step S141, based on the prediction model of the cross entropy loss function, and updating the weights and biases of the logistic binomial tree network model according to the feedforward network.

[0077] Step S142, predicting the current file to be detected according to the trained logical binomial tree network model, and obtaining the confidentiality detection result of the current file to be detected based on the word embedding vector of the current file to be detected.

[0078] Among them, the feedforward network is represented by two layers of linear transformation, namely the first layer linear transformation and the second layer linear transformation. The feedforward network is obtained by calling the activation function to process the weights and biases of the first layer linear transformation and the second layer linear transformation.

[0079] In a specific embodiment, the present invention provides a Tika-based file confidentiality detection implementation method. In order to specifically describe the technical defects mentioned in the background, a brief description is first given by combining code segment 1 and code segment 2 as an example.

[0080] Code Snippet 1:

[0081] File 11:

[0082] public class sdk_key{

[0083] private static string API_KEY="MyAPIKey12345"; / / Key

[0084] public boolean authenticate(String username,String password){

[0085] / / Logic for verifying user identity

[0086] }

[0087] }

[0088] File 12:

[0089] / / Pre-add the environment variable {Jupyter_python: C:\Users\xxx} to this server

[0090] String envPath=System.getenv("Jupyter_notebook");

[0091]

[0092] Code Snippet 2:

[0093] File 21:

[0094]

[0095] Document 22:

[0096] #Actually stored key API_KEY = 'sk-xxxxxxxH'

[0097] In this embodiment, code segment 1 is written in Java, and there are comments and variable names containing sensitive words such as "password" and "key", and the system detects hard-coded passwords. However, in fact, in file 11, the above API_KEY is a sample encrypted data used in the development environment. That is, in an actual engineering environment, real environment variables or configuration files will be used to load real encrypted data, as shown in file 12.

[0098] Code segment 2 is written in Python, which shows how to call the key code of the current artificial intelligence model for secondary development through an external application. It is not difficult to see that the meaning expressed by its code is highly similar to that of code segment 1. However, although its name also involves obviously confidential content such as API_KEY, it is essentially obtained through the key.env file, so whether it poses a security risk is actually difficult to determine only from file 21. Only by judging whether file 22 is encrypted can we determine whether it is confidential.

[0099] In this embodiment, both code segment 1 and code segment 2 are likely to be misjudged as containing sensitive information and presenting a security risk. In addition, code segments 1 and 2 correspond to code segment 3, which has a confidentiality risk.

[0100] Code Snippet 3:

[0101]

[0102] Similar to the above analysis, code segment 3 also contains sensitive information of the key. Although Tika will still determine that code segment 3 has a high probability of involving confidentiality risks, its logical process of deducing that code segment 3 has security risks is wrong.

[0103] It should be noted that, for the sake of convenience, only part of the above code is given. For example, taking code segment 2 as an example, it removes the irrelevant header files such as import os and import load_dotenv.

[0104] In this embodiment, we first need to explain the structural framework of Tika, which mainly includes:

[0105] Detector, used to detect the type of document and select the appropriate parser.

[0106] Parsers are used to parse documents of a specific type and extract content and metadata.

[0107] Metadata, an object used to store and transmit metadata information.

[0108] Content processors are used to process the content output by the parser and write the content to a string, file, or other output target.

[0109] The monitor and metadata are usually based on a fixed file format, and standardized detection data is sufficient. For the parser, the following two interfaces are mainly involved in this embodiment.

[0110] void Parser(InputStream stream,ContentHandler handler,Metadatametadata,ParseContext context)throws IOException,SAXException,TikaException;

[0111] public class ParseContext{

[0112] public static void main(String[]args)throws Exception{

[0113] / / Create an input stream to read the Word document

[0114] try(InputStream stream=new FileInputStream("11.docx")){

[0115] / / Create content handler

[0116] ContentHandler handler = new BodyContentHandler (-1); / / -1 means no limit on content size

[0117] / / Create metadata object

[0118] Metadata metadata=new Metadata();

[0119] / / Create parsing context

[0120] ParseContext context=new ParseContext();

[0121] / / Create the main parser

[0122] Parser parser=new AutoDetectParser();

[0123] / / Load sensitive word library

[0124] public static Set <string>loadSensitiveLexicon(){}

[0125] / / Detect sensitive words

[0126] public static List <string>detectSensitiveWords(String content,Set <string>sensitiveWords){}

[0127] / / Build a logical binomial tree

[0128] public static LogicalBinaryTree constructLogicalBinaryTree(List <string>detectedWords,String content){}

[0129] / / Calculate word embedding vector and similarity

[0130] public static void calculateWordEmbeddings(LogicalBinaryTreelogicalTree){}

[0131] / / Deep learning detection

[0132] public static DetectionResult deepLearningDetection(LogicalBinaryTreelogicalTree){}

[0133] / / Output the test results

[0134] public static void outputDetectionResult(DetectionResultdetectionResult){}

[0135] / / Define the class of logical binomial tree

[0136] static class LogicalBinaryTree{}

[0137] / / Execute parsing

[0138] parser.parse(stream,handler,metadata,context);

[0139] }

[0140] }

[0141] }

[0142] Among them, the Parser interface is the core interface of Tika, which defines the methods that all parsers must implement. For example, specific file format parsers such as PDFParser and TXTParser need to inherit this interface, which is used to parse the content in the input stream and extract text and metadata.

[0143] In this embodiment, the ParseContext class is used to pass additional information or services during the parsing process, which integrates the context information to achieve complete control of the parsed content. It is indispensable for the file confidentiality detection implementation method, especially for the security analysis of sensitive words.

[0144] In this embodiment, the present invention provides a method for realizing file confidentiality detection based on Tika, including steps 1 to 4:

[0145] Step 1: Lock sensitive words in text or code based on the sensitive information vocabulary.

[0146] For the files to be tested for confidentiality, use Parser to parse all files in its directory and convert all relevant information into strings, such as the above code segment 2. For sensitive words to be tested, relevant sensitive words can be extracted based on the pre-set sensitive word library, such as the characters of "interface key", "key.env" and "api_key" in code segment 2.

[0147] Step 2: Based on semantic analysis and combined with the context information of sensitive words, a logical binomial tree corresponding to the sensitive words is constructed.

[0148] Figure 2A to Figure 2C The logical binomial trees corresponding to code segments 1 to 3 are shown respectively. The reason for choosing the logical binomial tree is that the structure logic of the code is more complex. If the graph model is used, it is difficult to accurately describe the causal relationship between the codes, and it is also difficult to describe the logical nesting relationship between the codes. If the ordinary binary tree model is used, it is difficult to describe the logical nesting relationship between the codes.

[0149] In this embodiment, the causal relationship is represented in the form of parent-child nodes, and the logical nested relationship is represented in the form of sibling nodes, such as Figure 2A to Figure 2C shown.

[0150] exist Figure 2A In the code snippet 1, 01 to 10 represent sdk_key, API_KEY, "MyAPIKey12345", "API_KEY", API_KEY, properties, fileInput, keyPath, " / key.env", envPath, and "Jupyter_notebook" respectively.

[0151] exist Figure 2B In the code, 01 to 05 can represent load_dotenv in code segment 2,

[0152] "C: / Users / xxx / key.env",api_key,"API_KEY",API_KEY,'sk-xxxxxxxH'.

[0153] exist Figure 2C In the code, 00 to 04 can represent username, get_key, username+"_db_1234", username+"_api_key_888", and "8888" in code segment 3 respectively.

[0154] It should be noted that Figure 2A to Figure 2C The node x1, x2 or x3 in is a blank node, which itself has no meaning and is only used to determine the logical relationship between normal nodes and the distance value between normal nodes. In this embodiment, "blank node" is not mentioned, and all refer to non-blank normal nodes.

[0155] by Figure 2A As an illustration, the distance between nodes is considered as the distance between their positions in the text. The distance between a node and a blank node can be set arbitrarily, as long as the distance between nodes is accurate. For example, since the sum of the distance between node 00 and node x2 and the distance between node 01 and node x2 is a fixed value, the distance between any node 00 and node x2 or the distance between node 01 and node x2 can be set arbitrarily.

[0156] It is important to note that although the technical term used in this article is logical binomial tree, in fact Figures 2A to 2C It looks more like a Fibonacci tree than a binomial tree. But this is only because we did not fill in all the blank nodes! It should be noted that when calculating the jth leaf node in the i-th child binomial tree, the specific values ​​of i and j must be determined from the perspective of the binomial tree.

[0157] Step 3: Based on the word embedding model, calculate the word embedding vector corresponding to each node, and based on the word embedding vector, calculate the similarity between any two nodes; and based on the similarity, classify different nodes into types.

[0158] Specifically, the word embedding model can be Word2Vec or BERT. In the present invention, Word2Vec is selected to obtain the word embedding vector x corresponding to each node. j .

[0159] Node v k The associated adjacency matrix V k , whose expression is:

[0160]

[0161] In the formula, if i=j, v ij Represents the i-th child node of a vertex in a logical binomial tree. If i≠j, v ij Indicates the node v from the jth leaf node to the node in the binomial tree whose i-th child node is the root node k The distance between them is i,j=1,2,…,H. H represents the depth of the logical binomial tree.

[0162] by Figure 2A As an illustration: Since the depth of the sub-binomial tree with node x2 as the vertex is 2, it is actually the third sub-binomial tree of the vertex in the logical binomial tree. Correspondingly, leaf nodes 00 and 02 correspond to v 31 With v 32 Since the depth of the sub-binomial tree with node 03 as the vertex is 5, it is actually the sixth sub-binomial tree of the vertex in the logical binomial tree. At this time, leaf nodes 10, 08 and 05 correspond to v 65 、v 64 With v 62 .

[0163] by Figure 2B As an illustration: leaf nodes 00, 03, and 05 correspond to v 43 、v 42 With v 21 Note that the actual depth of the binomial tree with x2 as the root node is 3 instead of 2. In addition, node 01 is not a leaf node. A leaf node can be understood as the last (bottommost) node determined by a causal relationship rather than a logical nested relationship.

[0164] by Figure 2C As an illustration: leaf nodes 00 and 02 correspond to v 11 With v 41 . Note that the actual depth of the binomial tree with 01 as the root node is 3, not 1. In other words, the actual depth of the binomial tree is greater than or equal to the root node (for example: Figure 2C 01 in ), and is greater than or equal to the root node (for example: Figure 2B x2 in (for example: Figure 2B The number of child nodes of x3) in the root node + 1, and is greater than or equal to the number of child nodes of the child nodes of the child nodes of the root node + 2, and so on.

[0165] In this embodiment, we only focus on leaf nodes because only the final leaf nodes can truly determine the risk of security disclosure. This is true for both the environment variables in code segment 1 and the keys actually stored in code segment 2. Further analysis of code segment 2 shows that if only a logical binomial tree is constructed for file 21, its security risk is actually unknown. When file 22 is included, we actually throw (transfer) the security risk to file 22.

[0166] In this embodiment, steps A1 to A7 are also included:

[0167] Step A1, calculate the mapping vector of each node in the logical binomial tree according to the data type of the node.

[0168] Specifically, the data types are obtained by classifying different nodes according to similarity, and based on the values ​​corresponding to the nodes of the corresponding data type collected at any time, combined with the preset weight learning matrix, the mapping vector of each node in the logical binomial tree is calculated.

[0169] Step A2, calculating the neighborhood feature vector of any node with respect to any type of data information.

[0170] Specifically, based on a node set of the same data type in the logical binomial tree and an adjacency matrix associated with the nodes in the node set, a neighborhood feature vector of any node in the logical binomial tree with respect to any data type is calculated.

[0171] Step A3, calculate the attention scores of all data types.

[0172] Specifically, the activation function is called to calculate the attention scores of all data types based on the preset attention vector of any data type.

[0173] Step A4, based on the attention score, calculate the influence factor between any two nodes in the previous and next time.

[0174] Specifically, the influence factor between any two nodes is calculated based on the attention scores of all data types in the logical binomial tree.

[0175] Step A5: merging nodes with the same root based on a preset confidence threshold.

[0176] Specifically, when the influence factor between two nodes is greater than a preset confidence threshold (usually set to 0.95), the two nodes are merged to form a new node whose value is the sum of the original two nodes.

[0177] It should be noted that the confidence threshold should be related to the total number of nodes, that is, when the number of nodes is greater, the confidence threshold is relatively smaller, and when the total number of nodes is smaller, the confidence threshold is relatively larger.

[0178] Step A6, calculating a characteristic vector of each data type according to the influence factor between two nodes of the same type in the previous and next time periods.

[0179] Specifically, the activation function is called to process the influencing factors and mapping vectors obtained above, and the processing result of the activation function is weighted to obtain a weighted feature vector of the corresponding data type.

[0180] Step A7, building a heap embedding matrix, thereby building a logistic binomial tree network model.

[0181] Specifically, a logical binomial tree network model is first established according to the heap embedding matrix of the logical binomial tree, and the weight matrix in the logical binomial tree network model is initialized. The word embedding vector is converted into a query vector, a key vector, and a value vector through a self-attention layer.

[0182] Step 4: Based on the deep learning network, input the word embedding vector to obtain the detection result.

[0183] Specifically, it includes steps 4.1 to 4.2:

[0184] Step 4.1, based on the prediction model of the cross entropy loss function, the weights and biases of the logistic binomial tree network model are updated according to the feedforward network.

[0185] By iterating step 4.1 multiple times, a more robust logistic binomial tree network model is established.

[0186] Step 4.2, predict the current file to be detected according to the trained logical binomial tree network model, and obtain the confidentiality detection result of the current file to be detected based on the word embedding vector of the current file to be detected.

[0187] Among them, the feedforward network is represented by two layers of linear transformation, namely the first layer linear transformation and the second layer linear transformation. The feedforward network is obtained by calling the activation function to process the weights and biases of the first layer linear transformation and the second layer linear transformation.

[0188] The following is a description of a file confidentiality detection implementation device based on Tika provided by the present invention. The file confidentiality detection implementation device based on Tika described below and the file confidentiality detection implementation method based on Tika described above can be referenced to each other.

[0189] In one embodiment, a device for implementing file confidentiality detection based on Tika includes a sensitive word locking module, a logical binomial tree construction module, a node classification module, and a confidentiality detection module.

[0190] The sensitive word locking module is used to obtain a sensitive information word library, and detect and lock sensitive words in the target text or target code based on the sensitive information word library.

[0191] The logical binomial tree construction module is used to construct a logical binomial tree corresponding to sensitive words through semantic analysis according to the context information of sensitive words.

[0192] The node classification module is used to calculate the word embedding vector corresponding to each node in the logical binomial tree through the word embedding model, and calculate the similarity between any two nodes based on the word embedding vector. The similarity is used to classify different nodes.

[0193] The confidentiality detection module is used to use the word embedding vector as the input of the deep learning network model to call the deep learning network to detect the word embedding vector and obtain the confidentiality detection result of the target text or target code.

[0194] The relationship between the parent node and the child node in the logical binomial tree is used to represent the causal relationship of the context information of the sensitive words, and the relationship between the sibling nodes is used to represent the logical nested relationship of the context information of the sensitive words.

[0195] In this embodiment, the present invention provides a Tika-based file confidentiality detection implementation device, and the sensitive word locking module is specifically used to:

[0196] All sub-files under the file directory to be checked for confidentiality are parsed by Parser, and the parsing results are converted into strings. The parsing results include sensitive words in the file to be checked for confidentiality.

[0197] Based on the pre-entered sensitive information word library, sensitive words are extracted from the parsing results and locked.

[0198] In this embodiment, the present invention provides a Tika-based file confidentiality detection implementation device, and the node classification module is specifically used for:

[0199] The word embedding model is used to obtain the word embedding vector corresponding to each node in the logical binomial tree, and the word embedding vector corresponding to the node v in the logical binomial tree is obtained based on the word embedding vector. k The associated adjacency matrix V k .

[0200] The expression of the adjacency matrix is:

[0201]

[0202] In the formula, if i=j, v ij Represents the i-th child node of a vertex in a logical binomial tree. If i≠j, v ij Indicates that in the sub-binomial tree with the i-th child node as the root node, the j-th leaf node to the node v k The distance between them, i,j = 1, 2,…, H, where H represents the depth of the logical binomial tree.

[0203] In this embodiment, the present invention provides a Tika-based file confidentiality detection implementation device, which further includes a first computing module, which is used to:

[0204] The data types are obtained by classifying different nodes according to their similarities, and based on the values ​​corresponding to the nodes of the corresponding data types collected at any time, combined with the preset weight learning matrix, the mapping vector of each node in the logical binomial tree is calculated.

[0205] Based on a set of nodes of the same data type in the logical binomial tree and an adjacency matrix associated with the nodes in the node set, a neighborhood feature vector of any node in the logical binomial tree with respect to any data type is calculated.

[0206] In this embodiment, the present invention provides a Tika-based file confidentiality detection implementation device, which further includes a second computing module, which is used to:

[0207] Call the activation function to calculate the attention scores of all data types based on the preset attention vector of any data type.

[0208] Calculate the influence factor between any two nodes based on the attention scores of all data types in the logistic binomial tree.

[0209] In this embodiment, the present invention provides a Tika-based file confidentiality detection implementation device, which also includes a model building and initialization module for:

[0210] Based on the preset confidence threshold, the child nodes of the same data type are merged, and according to the influence factor between any two nodes of the same data type, the activation function is called to calculate the feature vector of each data type.

[0211] A logical binomial tree network model is established according to the heap embedding matrix of the logical binomial tree, and the weight matrix in the logical binomial tree network model is initialized. The word embedding vector is converted into a query vector, a key vector, and a value vector through the self-attention layer.

[0212] In this embodiment, the present invention provides a Tika-based file confidentiality detection implementation device, and the confidentiality detection module is specifically used for:

[0213] The prediction model is based on the cross entropy loss function, and the weights and biases of the logistic binomial tree network model are updated according to the feed-forward network.

[0214] The current file to be detected is predicted according to the trained logical binomial tree network model, and the confidentiality detection result of the current file to be detected is obtained based on the word embedding vector of the current file to be detected.

[0215] Among them, the feedforward network is represented by two layers of linear transformation, namely the first layer linear transformation and the second layer linear transformation. The feedforward network is obtained by calling the activation function to process the weights and biases of the first layer linear transformation and the second layer linear transformation.

[0216] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0217] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0218] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0219] The computer program instructions for performing the disclosed operation may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit may be customized by utilizing the state information of a computer-readable program instruction, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0220] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0221] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0222] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0223] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.< / string> < / string> < / string> < / string>

Claims

1. A method for realizing file confidentiality detection based on Tika, characterized in that: The method comprises: Acquire a sensitive information word library, and detect and lock sensitive words in the target text or target code based on the sensitive information word library; According to the context information of the sensitive words, construct a logical binomial tree corresponding to the sensitive words through semantic analysis; Calculate the word embedding vector corresponding to each node in the logical binomial tree through the word embedding model, and calculate the similarity between any two nodes based on the word embedding vector, wherein the similarity is used to classify different nodes; Using the word embedding vector as an input of a deep learning network model to call the deep learning network to detect the word embedding vector and obtain a confidentiality detection result of the target text or target code; The relationship between the parent node and the child node in the logical binomial tree is used to represent the causal relationship of the context information of the sensitive words, and the relationship between the sibling nodes is used to represent the logical nested relationship of the context information of the sensitive words.

2. The method for realizing file confidentiality detection based on Tika according to claim 1, characterized in that: The step of obtaining a sensitive information word library and detecting and locking sensitive words in a target text or target code based on the sensitive information word library includes: Parse all subfiles in the file directory to be tested for confidentiality by Parser, and convert the parsing results into strings, wherein the parsing results include sensitive words in the file to be tested for confidentiality; The sensitive words are extracted from the analysis results based on the pre-input sensitive information word library, and the sensitive words are locked.

3. The method for realizing file confidentiality detection based on Tika according to claim 1, characterized in that: The word embedding model is used to calculate the word embedding vector corresponding to each node in the logical binomial tree, and the similarity between any two nodes is calculated based on the word embedding vector, and the similarity is used to classify different nodes, including: The word embedding vector corresponding to each node in the logical binomial tree is obtained through the Word2Vec word embedding model, and the word embedding vector corresponding to the node v in the logical binomial tree is obtained based on the word embedding vector. k The associated adjacency matrix V k ; Wherein, the expression of the adjacency matrix is: In the formula, if i=j, v ij Represents the i-th child node of a vertex in a logical binomial tree. If i≠j, v ij In the sub-binomial tree whose root node is the i-th child node, the j-th leaf node to the node v k The distance between them, i,j = 1, 2,…, H, where H represents the depth of the logical binomial tree.

4. The method for realizing file confidentiality detection based on Tika according to claim 3 is characterized in that: The word embedding model is used to calculate the word embedding vector corresponding to each node in the logical binomial tree, and the similarity between any two nodes is calculated based on the word embedding vector, and the similarity is used to classify different nodes, and then it also includes: According to the data types obtained by classifying different nodes according to similarity, and based on the values ​​corresponding to the nodes of the corresponding data types collected at any time, combined with the preset weight learning matrix, the mapping vector of each node in the logical binomial tree is calculated; and Based on a node set of the same data type in the logical binomial tree and an adjacency matrix associated with nodes in the node set, a neighborhood feature vector of any node in the logical binomial tree with respect to any data type is calculated.

5. The method for realizing file confidentiality detection based on Tika according to claim 4 is characterized in that: The word embedding model is used to calculate the word embedding vector corresponding to each node in the logical binomial tree, and the similarity between any two nodes is calculated based on the word embedding vector, and the similarity is used to classify different nodes, and then it also includes: Call the activation function to calculate the attention scores of all data types based on the preset attention vector of any data type; According to the attention scores of all data types in the logical binomial tree, the influence factor between any two nodes is calculated.

6. The method for realizing file confidentiality detection based on Tika according to claim 5, characterized in that: The word embedding model is used to calculate the word embedding vector corresponding to each node in the logical binomial tree, and the similarity between any two nodes is calculated based on the word embedding vector, and the similarity is used to classify different nodes, and then it also includes: Based on a preset confidence threshold, child nodes of the same data type are merged, and according to the influence factor between any two nodes of the same data type, an activation function is called to calculate the feature vector of each data type; and A logical binomial tree network model is established according to the heap embedding matrix of the logical binomial tree, and the weight matrix in the logical binomial tree network model is initialized, and the word embedding vector is converted into a query vector, a key vector and a value vector through a self-attention layer.

7. The method for realizing file confidentiality detection based on Tika according to claim 6, characterized in that: The using the word embedding vector as the input of the deep learning network model to call the deep learning network to detect the word embedding vector and obtain the confidentiality detection result of the target text or target code includes: A prediction model based on a cross entropy loss function, and updating the weights and biases of the logistic binomial tree network model according to a feedforward network; Predicting the current file to be detected according to the trained logical binomial tree network model, and obtaining the confidentiality detection result of the current file to be detected based on the word embedding vector of the current file to be detected; Among them, the feedforward network is represented by two layers of linear transformation, namely the first layer of linear transformation and the second layer of linear transformation. The feedforward network is obtained by calling the activation function to process the weights and biases of the first layer of linear transformation and the second layer of linear transformation.

8. A file confidentiality detection implementation device based on Tika, characterized in that: The device comprises: A sensitive word locking module, used to obtain a sensitive information word library, and detect and lock sensitive words in the target text or target code based on the sensitive information word library; A logical binomial tree construction module, used to construct a logical binomial tree corresponding to the sensitive words through semantic analysis according to the context information of the sensitive words; A node classification module, used to calculate the word embedding vector corresponding to each node in the logical binomial tree through a word embedding model, and calculate the similarity between any two nodes based on the word embedding vector, wherein the similarity is used to classify different nodes; A confidentiality detection module, used to use the word embedding vector as an input of a deep learning network model to call the deep learning network to detect the word embedding vector and obtain a confidentiality detection result of the target text or target code; The relationship between the parent node and the child node in the logical binomial tree is used to represent the causal relationship of the context information of the sensitive words, and the relationship between the sibling nodes is used to represent the logical nested relationship of the context information of the sensitive words.

9. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images