A sensitive information identification method and system

Abstract

The embodiment of the application provides a sensitive information recognition method and system, comprising: obtaining to-be-published information, extracting a segment of an expression entity from the to-be-published information based on a dictionary tree of sensitive information, generating a candidate entity pair of the segment of the expression entity and a corresponding entity in the dictionary tree; inputting an entity matching model to perform matching calculation on the segment of the expression entity and the corresponding entity in the candidate entity pair, and outputting a matching value; taking the candidate entity pair whose matching value meets a preset configuration threshold as a target candidate entity pair, obtaining a category of the entity corresponding to the segment of the expression entity based on a pre-constructed multi-modal knowledge graph of sensitive information; determining a recognition result of the to-be-published information based on the category of the entity, and recalling the to-be-published information. Through the multi-modal knowledge graph, the connection relationship between the segment of the expression entity and the entity layer is realized, the recognition problem of the to-be-published information is mapped into an entity linking problem, and the recognition difficulty caused by the data sparsity of the sensitive information is effectively alleviated.

Description

Technical Field

[0001] This invention relates to the field of information processing, and specifically to a method and system for identifying sensitive information. Background Technology

[0002] With the development of the Internet, there are often malicious users who deliberately publish sensitive information and content online. For example, politically sensitive information often has extremely high social harm and involves multiple aspects of identification, such as political events and political figures. It has the characteristics of wide coverage, a lot of identification content, and strong correlation between events and figures.

[0003] Currently, common methods for identifying sensitive information use deep learning algorithms to classify and detect sensitive content in published texts. However, the model training effect depends entirely on the corpus features, making it difficult to effectively improve the identification coverage of certain types of sensitive information that are sparsely distributed and highly harmful. Summary of the Invention

[0004] This invention provides a sensitive information identification method and system to address the problem that model training performance relies entirely on corpus features, making it difficult to effectively improve the identification coverage of a certain type of sensitive information that is sparsely distributed and highly harmful.

[0005] To achieve the above objectives, in one aspect, embodiments of the present invention provide a method for identifying sensitive information, comprising:

[0006] Obtain the information to be published, extract fragments of express entities from the information to be published based on the trie of sensitive information, and generate candidate entity pairs by combining the fragments of express entities with the corresponding entities in the trie. The trie includes entities and their pronouns, and the entity refers to entity text keywords representing sensitive information or the corresponding image of the entity text keywords.

[0007] The candidate entity pair input entity matching model performs matching calculations on the fragments of the entity expressed in the candidate entity pair and the corresponding entities, and outputs the matching values ​​of the fragments of the entity expressed in the candidate entity pair and the corresponding entities.

[0008] Candidate entity pairs whose matching values ​​meet the preset configuration threshold are taken as target candidate entity pairs. Based on the pre-constructed multimodal knowledge graph of sensitive information, the categories of entities corresponding to the fragments of the expressed entities in the target candidate entity pairs are obtained.

[0009] The identification result of the information to be published is determined based on the category of the entity, and the information to be published is recalled.

[0010] On the other hand, embodiments of the present invention provide a sensitive information identification system, including:

[0011] The traversal unit is used to obtain the information to be published, extract fragments of the expressive entity from the information to be published based on the trie of sensitive information, and generate candidate entity pairs by combining the fragments of the expressive entity with the corresponding entities in the trie. The trie includes entities and their pronouns. The entity refers to the entity text keyword representing sensitive information or the corresponding image of the entity text keyword.

[0012] An entity matching model is used to input the candidate entity pair into the entity matching model, perform matching calculations on the fragments expressing the entity in the candidate entity pair and the corresponding entities, and output the matching values ​​between the fragments expressing the entity in the candidate entity pair and the corresponding entities.

[0013] The category determination unit is used to take the candidate entity pairs whose matching values ​​meet the preset configuration threshold as target candidate entity pairs, and obtain the category of the entity corresponding to the fragment of the expressed entity in the target candidate entity pair based on the pre-constructed multimodal knowledge graph of sensitive information.

[0014] The recall unit is used to determine the identification result of the information to be published based on the category of the entity, and recall the information to be published.

[0015] The above technical solution has the following beneficial effects: For information to be published, fragments of expressive entities are extracted from the information based on a trie of sensitive information; these fragments are then used to generate candidate entity pairs with corresponding entities in the trie; a pre-constructed multi-modal knowledge graph of sensitive information is used to determine the categories of expressive entity fragments within the target candidate entity pairs whose matching values ​​meet a preset configuration threshold; and the information to be published is then recalled. This achieves the identification of scattered entities, effectively alleviating the identification difficulties caused by the data sparsity of sensitive information. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a sensitive information identification method according to an embodiment of the present invention;

[0018] Figure 2 This is a structural diagram of the sensitive information identification system according to an embodiment of the present invention;

[0019] Figure 3 This is the overall identification architecture of this invention embodiment;

[0020] Figure 4Example of an embodiment of the present invention: hierarchical results of a political multimodal knowledge graph;

[0021] Figure 5 This is the multi-modal knowledge graph entity embedding training according to an embodiment of the present invention;

[0022] Figure 6 This is a flowchart illustrating the entity linking process according to an embodiment of the present invention;

[0023] Figure 7 This is a structural diagram of the entity matching model according to an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] like Figure 1 As shown, in conjunction with embodiments of the present invention, a sensitive information identification method is provided. This method constructs a multi-modal knowledge graph, determines the matching degree between fragments of the entity expressing the information and the entity itself for the information to be published, and inputs target candidate entity pairs whose matching values ​​meet a preset threshold into the pre-constructed multi-modal knowledge graph of sensitive information. This yields the category of the entity corresponding to the fragment of the entity expressing the information in the target candidate entity pair. This effectively alleviates the data sparsity problem of sensitive information and simultaneously improves the identification coverage of any type of sensitive information, thereby enabling the recall of information to be published that belongs to the sensitive information category. The sensitive information identification method includes:

[0026] S301: Obtain the information to be published, extract the fragments of the express entities from the information to be published based on the trie of sensitive information, and generate candidate entity pairs by combining the fragments of the express entities with the corresponding entities in the trie. The trie includes entities and their pronouns. The entity refers to the entity text keyword representing sensitive information or the image corresponding to the entity text keyword.

[0027] S302: The candidate entity pair is input to the entity matching model to perform matching calculations on the fragments of the entity expressed in the candidate entity pair and the corresponding entities, and the matching values ​​of the fragments of the entity expressed in the candidate entity pair and the corresponding entities are output.

[0028] S303: The candidate entity pairs whose matching values ​​meet the preset configuration threshold are taken as target candidate entity pairs. Based on the pre-constructed multimodal knowledge graph of sensitive information, the categories of entities in the target candidate entity pairs that correspond to the fragments of the expressed entities are obtained.

[0029] S304: Determine the identification result of the information to be published based on the category of the entity, and recall the information to be published.

[0030] Preferably, the multimodal knowledge graph of the sensitive information includes a category layer, an entity layer, and an image hosting layer, wherein the entity layer is associated with the category layer, and the image hosting layer is associated with the entity layer;

[0031] S305: The multimodal knowledge graph of the sensitive information is constructed through the following methods, including:

[0032] S3051: Collect category keywords for sensitive information of a set category, create a category tree structure based on the category keywords to obtain a category layer, the tree structure includes leaf categories, and the leaf categories are used to directly connect with the entity layer;

[0033] S3052: Identify the entity category to which words in the text containing the defined sensitive information belong, obtain entity text keywords for each entity category, construct triples of entity text keywords using entity text keywords, and construct an entity layer using triples of entity text keywords; wherein, the triples of entity text keywords include: head entity, relation, and tail entity, the head entity represents a first entity text keyword, the tail entity represents a second entity text keyword that has an association relationship with the head entity, and the relation represents the association relationship between the tail entity and the head entity;

[0034] S3053: Associate the head entity of the entity layer with the leaf category of the category layer;

[0035] S3054: Based on the entity text keywords of sensitive information of a set category, obtain the corresponding images and their structured information from the dataset. Construct triples of the images using the obtained images, and construct an image bed layer using the triples of each image. The triples of the images include: head entity, relation, and tail entity. The tail entity is the image. The head entity is the head entity or tail entity of the entity layer. The relation is the structured information of the image. The structured information of the image is used to represent the association relationship between the head entity and the tail entity.

[0036] Preferably, the sensitive information identification method further includes:

[0037] S306: Based on the entities in the entity layer or image hosting layer within the multimodal knowledge graph of the sensitive information, construct a trie for the sensitive information. The trie includes entities and their pronouns. The entities include the head entity and tail entity in the triples of entity text keywords, and the tail entity in the triples of images.

[0038] Preferably, before constructing the entity layer using triples of each entity's text keywords, the method further includes:

[0039] S3055: For the constructed triples of entity text keywords, based on the similarity between entity text keywords, different entity text keywords in the head entity whose similarity exceeds a set similarity threshold are represented by the same node identifier, different entity text keywords in the tail entity whose similarity exceeds a set similarity threshold are represented by the same node identifier, and different relations with the same node identifier in the relation are represented by the same node identifier; triples of entity text keywords that do not belong to the set category of sensitive information and triples of erroneous entity text keywords are removed, and the remaining triples of entity text keywords are used to construct the entity layer;

[0040] Before constructing the image layer using image triples, the method further includes:

[0041] S3056: For the triples of each constructed image, remove the triples of images containing sensitive information that do not belong to the set category, as well as the triples of erroneous images. The triples of the retained images are used to construct the image hosting layer.

[0042] Preferably, it further includes:

[0043] S307: After constructing the entity layer using triples of text keywords of each entity, traverse the triples of text keywords of each entity in the entity layer, wherein the head entity is represented by the first entity text keyword and the descriptive information of the entity, the tail entity is represented by the second entity text keyword and the descriptive information of the entity, and the relationship is represented by text.

[0044] S308: Input the head entity, tail entity, and relation in the triplet of the entity text keyword into the pre-trained text feature extraction network in sequence. Extract the text feature vectors of the head entity, tail entity, and relation through the pre-trained text feature extraction network, output the text feature vector of the triplet of the entity text keyword, and save the text feature vector of the triplet of each entity text keyword to the entity vector table.

[0045] S309: The pre-trained text feature extraction network is trained using the following method:

[0046] S3091: Input the head entities, relations, and tail entities of the training samples into the convolutional network of the general knowledge base transE and the Robert language model for iterative model training until the loss function of the text feature extraction network reaches the model convergence condition, thus obtaining the pre-trained text feature extraction network; the specific steps for iterative model training include:

[0047] S3092: Input the head entity, relation, and tail entity of the training sample into the convolutional network of the general knowledge base transE to obtain their respective low-dimensional vectors of text features;

[0048] S3093: Input the low-dimensional vectors of text features of the head entity, relation, and tail entity into the Robert language model, hide the characters in the low-dimensional vectors of text features of the head entity, relation, and tail entity using the first preset probability, and randomly replace the characters in the low-dimensional vectors of text features of the head entity, relation, and tail entity using the second preset probability, and output the processed low-dimensional vectors of text features of the head entity, relation, and tail entity.

[0049] S3094: The loss function of the text feature extraction network includes the loss function of the general knowledge base transE and the loss function of the Robert language model. The loss function of the general knowledge base transE represents the relationship between the sum of the low-dimensional text feature vectors of the head entity and the low-dimensional text feature vectors of the relation and the low-dimensional text feature vectors of the tail entity. The loss function of the Robert language model represents the cross-entropy of the low-dimensional text feature vectors of the head entity, relation, and tail entity after processing.

[0050] Preferably, in step 301, the process of extracting fragments of expressive entities from the information to be published using the trie based on sensitive information, and generating candidate entity pairs by combining the fragments of expressive entities with the corresponding entities in the trie, includes:

[0051] The information to be published is traversed through the trie of the sensitive information. Based on the entities in the trie, fragments of the entities to be published and their positions are extracted. The fragments of the entities are then paired with the corresponding entities in the trie to generate candidate entity pairs.

[0052] In step 302, the step of using the candidate entity pair input entity matching model to perform matching calculations on the fragments representing entities in the candidate entity pair with the corresponding entities, and outputting the matching values ​​between the fragments representing entities in the candidate entity pair and the corresponding entities, includes:

[0053] S3021: If the information to be published does not contain images, a pre-trained text feature extraction network and a pre-trained text matching network are used as the entity matching model, specifically including:

[0054] S3021-1: Input the information to be published into the pre-trained text feature extraction network to extract the low-dimensional vector of each character of the information to be published; input the low-dimensional vector of the character at the corresponding position of the segment representing the entity into the pre-trained text feature extraction network to output the expression vector corresponding to the segment representing the entity.

[0055] S3021-2: Extract entity vectors corresponding to the segments representing the entities from the entity vector table;

[0056] S3021-3: Input the expression vector of the entity segment and the corresponding entity vector into the pre-trained text matching network for matching prediction to obtain the first text matching value of the information to be published;

[0057] S3021-4: Use the first text matching value as the matching value between the fragment representing the entity and the corresponding entity in the candidate entity pair;

[0058] S3022: If the information to be published contains images, a decoupled pre-trained text feature extraction network and a pre-trained image feature extraction network, as well as a pre-trained text matching network and a pre-trained image matching network, are used as the entity matching model, specifically including:

[0059] S3022-1: Input the text information in the information to be published into a pre-trained text feature extraction network to extract the low-dimensional vector of each character in the text information; input the low-dimensional vector of the character at the corresponding position of the segment representing the entity into the pre-trained text feature extraction network to output the expression vector corresponding to the segment representing the entity.

[0060] S3022-2: Extract entity vectors corresponding to the segments representing the entities from the entity vector table;

[0061] S3022-3: Input the expression vector of the entity segment and the entity vector corresponding to the entity segment into the pre-trained text matching network for matching prediction to obtain the second text matching value of the information to be published;

[0062] as well as,

[0063] S3022-4: Input the information to be published into a pre-trained image feature extraction network to extract image features within the information to be published, and obtain the first image feature vector within the information to be published;

[0064] S3022-5: Input the corresponding entities of the images contained in the information to be published into a pre-trained image feature extraction network, extract the head entity vector and the relationship vector between the head entity and the tail entity, and use the vector composed of the head entity vector and the relationship vector as the second image feature vector of the corresponding entity of the image.

[0065] S3022-6: Input the first image feature vector and the second image feature vector into a pre-trained image matching network to obtain the image matching value of the information to be published;

[0066] The second text matching value and image matching value are used as the matching values ​​between the fragment representing the entity and the corresponding entity in the candidate entity pair.

[0067] Preferably, S310: The method for determining the target candidate entity pair includes:

[0068] S3101: If the information to be published does not contain an image, and the first text matching value is greater than the first text configuration threshold, the candidate entity pair is determined to be the target candidate entity pair.

[0069] S3102: If the information to be published contains an image, and the second text matching value is greater than the second text configuration threshold and the image matching value is greater than the image configuration threshold, the candidate entity pair is determined to be the target candidate entity pair; wherein, the first text configuration threshold is greater than the second text configuration threshold.

[0070] Preferably, in step 303, the multi-modal knowledge graph based on pre-constructed sensitive information obtains the categories of entities corresponding to the fragments of the expressed entities in the target candidate entity pairs, including:

[0071] The graph database is used to find the category of the corresponding entity in the target candidate pair in the multimodal knowledge graph of the sensitive information;

[0072] In step 304, determining the identification result of the information to be published based on the category of the entity includes:

[0073] The category, entity, and location of the fragment representing the entity are used as the identification results of the information to be published.

[0074] like Figure 2 As shown, in conjunction with embodiments of the present invention, a sensitive information identification system is provided, comprising:

[0075] Traversal unit 21 is used to obtain information to be published, extract fragments of express entities from the information to be published based on the trie of sensitive information, and generate candidate entity pairs by combining the fragments of express entities with the corresponding entities in the trie. The trie includes entities and their pronouns. The entity refers to entity text keywords representing sensitive information or the corresponding image of the entity text keywords.

[0076] Entity matching model 22 is used to perform matching calculations on the fragments representing entities in the input candidate entity pairs and the corresponding entities, and output the matching values ​​between the fragments representing entities in the candidate entity pairs and the corresponding entities.

[0077] The category determination unit 23 is used to take the candidate entity pairs whose matching values ​​meet the preset configuration threshold as target candidate entity pairs, and obtain the category of the entity corresponding to the fragment of the expressed entity in the target candidate entity pair based on the pre-constructed multimodal knowledge graph of sensitive information.

[0078] The recall unit 24 is used to determine the identification result of the information to be published based on the category of the entity, and recall the information to be published.

[0079] Preferably, the multimodal knowledge graph of the sensitive information includes a category layer, an entity layer, and an image hosting layer, wherein the entity layer is associated with the category layer, and the image hosting layer is associated with the entity layer;

[0080] A multimodal knowledge graph 25 for sensitive information is constructed by building category layer construction unit 251, entity layer construction unit 252, and image hosting layer construction unit 253, wherein:

[0081] Category layer construction unit 251 is used to collect category keywords of sensitive information of a set category, and create a tree structure of categories based on the category keywords to obtain a category layer. The tree structure includes leaf categories, which are used to be directly connected to the entity layer.

[0082] Entity layer construction unit 252 is used to identify the entity category to which words in text containing the defined sensitive information belong, obtain entity text keywords for each entity category, construct triples of entity text keywords, and construct the entity layer through the triples of entity text keywords; wherein, the triples of entity text keywords include: head entity, relation, and tail entity, the head entity represents a first entity text keyword, the tail entity represents a second entity text keyword that has an association with the head entity, and the relation represents the association relationship between the tail entity and the head entity; and

[0083] Associate the head entity of the entity layer with the leaf category of the category layer;

[0084] Image hosting layer construction unit 253 is used to obtain the corresponding images and their structured information from the dataset based on the entity text keywords of the sensitive information of the set category, construct triples of the images using the obtained images, and construct the image hosting layer through the triples of each image; wherein, the triples of the images include: head entity, relation, and tail entity, the tail entity is the image, the head entity is the head entity or tail entity of the entity layer, the relation is the structured information of the image, and the structured information of the image is used to represent the association relationship between the head entity and the tail entity.

[0085] Preferably, it further includes:

[0086] The trie construction unit 26 is used to construct a trie for sensitive information based on entities in the entity layer or image hosting layer within the multimodal knowledge graph of the sensitive information. The trie includes entities and their pronouns. The entities include the head entity and tail entity in the triple of entity text keywords, and the head entity and tail entity in the triple of images.

[0087] Preferably, it further includes:

[0088] The first processing unit 254 is configured to, before constructing the entity layer using triples of entity text keywords, represent different entity text keywords in the head entity with a similarity exceeding a set similarity threshold using the same node identifier, and represent different entity text keywords in the tail entity with a similarity exceeding a set similarity threshold using the same node identifier, and represent different relations with the same node identifier in the relation using the same node identifier, based on the similarity between entity text keywords; remove triples of entity text keywords containing sensitive information that do not belong to the set category, as well as triples of erroneous entity text keywords, and use the retained triples of entity text keywords to construct the entity layer;

[0089] The second processing unit 255 is used to, before constructing the image layer using image triples, remove image triples that do not belong to the set category of sensitive information and erroneous image triples from the already constructed image triples, and the retained image triples are used to construct the image hosting layer.

[0090] Preferably, it also includes an entity text keyword conversion unit 27, specifically used for:

[0091] After constructing the entity layer using triples of entity text keywords, the triples of entity text keywords within the entity layer are traversed. The head entity is represented by the first entity text keyword and the descriptive information of the entity, the tail entity is represented by the second entity text keyword and the descriptive information of the entity, and the relationship is represented by text.

[0092] The head entity, tail entity, and relation in the triplet of the entity text keyword are sequentially input into a pre-trained text feature extraction network. The pre-trained text feature extraction network extracts the text feature vectors of the head entity, tail entity, and relation respectively, and outputs the text feature vector of the triplet of the entity text keyword. The text feature vector of the triplet of each entity text keyword is saved to the entity vector table.

[0093] The sensitive information identification system further includes: a training unit 28 for a pre-trained text feature extraction network, specifically used for:

[0094] The head entities, relations, and tail entities of the training samples are input into the convolutional network of the general knowledge base transE and the Robert language model for iterative training until the loss function of the text feature extraction network reaches the model convergence condition, thus obtaining the pre-trained text feature extraction network. The specific steps for iterative model training include:

[0095] The head entities, relations, and tail entities of the training samples are input into the convolutional network of the general knowledge base transE to obtain their respective low-dimensional vectors of text features.

[0096] Input the low-dimensional vectors of text features of the head entity, relation, and tail entity into the Robert language model. Use the first preset probability to hide the characters in the low-dimensional vectors of text features of the head entity, relation, and tail entity, and use the second preset probability to randomly replace the characters in the low-dimensional vectors of text features of the head entity, relation, and tail entity. Output the processed low-dimensional vectors of text features of the head entity, relation, and tail entity.

[0097] The loss function of the text feature extraction network includes the loss function of the general knowledge base transE and the loss function of the Robert language model. The loss function of the general knowledge base transE represents the relationship between the sum of the low-dimensional text feature vectors of the head entity and the low-dimensional text feature vectors of the relation and the low-dimensional text feature vectors of the tail entity. The loss function of the Robert language model represents the cross-entropy of the low-dimensional text feature vectors of the head entity, relation and tail entity after processing.

[0098] Preferably, the traversal unit 21 is specifically used for:

[0099] The information to be published is traversed through the trie of the sensitive information. Based on the entities in the trie, fragments of the entities to be published and their positions are extracted. The fragments of the entities are then paired with the corresponding entities in the trie to generate candidate entity pairs.

[0100] The entity matching model 22 includes a text matching subunit 221 and an image matching subunit 222, wherein:

[0101] The text matching subunit 221 is used to use a pre-trained text feature extraction network and a pre-trained text matching network as entity matching models if the information to be published does not contain images.

[0102] The information to be published is input into a pre-trained text feature extraction network to extract the low-dimensional vector of each character in the information to be published; the low-dimensional vectors of the characters at the corresponding positions of the segments representing entities are input into the pre-trained text feature extraction network to output the expression vectors corresponding to the segments representing entities.

[0103] Extract entity vectors corresponding to the segments representing the entities from the entity vector table;

[0104] The expression vector of the entity segment and the corresponding entity vector are input into a pre-trained text matching network for matching prediction to obtain the first text matching value of the information to be published.

[0105] The first text matching value is used as the matching value between the fragment representing the entity and the corresponding entity in the candidate entity pair;

[0106] Image matching subunit 222 is used to use a decoupled pre-trained text feature extraction network and pre-trained image feature extraction network, as well as a pre-trained text matching network and pre-trained image matching network as entity matching models if the information to be published contains images.

[0107] The text information in the information to be published is input into a pre-trained text feature extraction network to extract the low-dimensional vector of each character in the text information; the low-dimensional vectors of the characters at the corresponding positions of the segments representing entities are input into the pre-trained text feature extraction network to output the expression vectors corresponding to the segments representing entities.

[0108] Extract entity vectors corresponding to the segments representing the entities from the entity vector table;

[0109] The expression vector of the entity segment and the entity vector corresponding to the entity segment are input into a pre-trained text matching network for matching prediction to obtain the second text matching value of the information to be published.

[0110] as well as,

[0111] The information to be published is input into a pre-trained image feature extraction network to extract image features within the information to be published, thus obtaining the first image feature vector within the information to be published.

[0112] The entities corresponding to the images contained in the information to be published are input into a pre-trained image feature extraction network to extract the head entity vector and the relationship vector between the head entity and the tail entity. The vector composed of the head entity vector and the relationship vector is used as the second image feature vector of the corresponding entity of the image.

[0113] The first image feature vector and the second image feature vector are input into a pre-trained image matching network to obtain the image matching value of the information to be published.

[0114] The second text matching value and image matching value are used as the matching values ​​between the fragment representing the entity and the corresponding entity in the candidate entity pair.

[0115] Preferably, it further includes a target candidate entity pair determination unit 28, specifically used for:

[0116] If the information to be published does not contain an image, and the first text matching value is greater than the first text configuration threshold, the candidate entity pair is determined to be the target candidate entity pair.

[0117] If the information to be published contains an image, and the second text matching value is greater than the second text configuration threshold and the image matching value is greater than the image configuration threshold, the candidate entity pair is determined to be the target candidate entity pair; wherein, the first text configuration threshold is greater than the second text configuration threshold.

[0118] Preferably, the category determination unit 23 is specifically used for:

[0119] The candidate entity pairs whose matching values ​​satisfy the preset configuration values ​​are taken as target candidate entity pairs, and the category of the corresponding entity in the target candidate entity pair is found in the multimodal knowledge graph of the sensitive information through the graph database;

[0120] The recall unit 24 is specifically used for:

[0121] The category, entity, and location of the fragment representing the entity are used as the identification results of the information to be published.

[0122] The technical solutions of the present invention will be described in detail below with reference to specific application examples. For technical details not described in the implementation process, please refer to the relevant descriptions above.

[0123] This invention discloses a sensitive information identification method based on a multimodal knowledge graph, which relates to the technical fields of sensitive information identification, natural language processing, and knowledge graphs. It can solve the shortcomings of existing technologies in the identification of any type of sensitive information. Based on a multimodal knowledge graph, this invention combines key technologies such as entity embedding, entity linking, and multimodal fusion to effectively improve the identification coverage of any type of sensitive information and has good interpretability.

[0124] Taking a multi-modal knowledge graph-based method for identifying politically sensitive information as an example, this paper constructs a multi-modal knowledge graph for the political domain based on expert domain knowledge and publicly available databases such as Wikipedia. This multi-modal knowledge graph mainly includes a category layer, an entity layer, and an image hosting layer. The category layer and entity layer establish connections, mapping the sensitive content classification and identification problem to an entity linking problem. When the input content contains category-related entities, the input content is recalled. This method improves the interpretability of the model by constructing the multi-modal knowledge graph through experts and mapping the classification and identification problem to an entity linking problem. It effectively utilizes the relationships between people and events to enhance recognition capabilities. Furthermore, by identifying scattered entities associated with categories, it effectively alleviates the data sparsity problem of sensitive information.

[0125] The overall implementation steps of this invention are as follows: Figure 3 As shown. Step 1: First, a multimodal knowledge graph is constructed by combining expert knowledge, NLP, web crawling, and other technologies. This multimodal knowledge graph establishes connections between category and entity layers, mapping the sensitive content classification and identification problem to an entity linking problem. Step 2: For the multimodal knowledge graph, embedding vectors of nodes and edges are trained by combining text feature extraction networks, image feature extraction networks, and the graph structure. Step 3: An entity linking dataset is constructed, and entity linking and entity matching models are trained. Finally, the multimodal strategy is combined to complete online recognition and prediction.

[0126] Step 1: Construction of a Multi-Modal Knowledge Graph

[0127] The process for constructing a multi-modal knowledge graph for political applications is attached. Figure 4 As shown,

[0128] Step 101: First, collect category keywords for sensitive information in the defined categories. Specifically, domain experts will sort out the category hierarchy: the category layer is a tree structure, and the leaf categories are the last layer of the category tree structure, directly connecting the leaf categories to the entity layer.

[0129] Step 102: Identify the entity category to which words in the text containing the defined sensitive information belong, and obtain the entity text keywords for each entity category. For example, the main method for collecting political entity text keywords is: 1. Perform entity recognition, word segmentation, stop word removal, and manual data cleaning on political text (data level) to filter 15 categories of entity text keywords, including people, events, systems, organizations, film and television works, positions, and symbols, forming a political category hierarchical structure; 2. Domain experts collect keywords based on people and events, organized chronologically, to form entity text keywords in the entity layer. Entity text keywords are used to construct triples of entity text keywords, and the entity layer is constructed through these triples. Each triple of entity text keywords includes: head entity, relation, and tail entity. The head entity represents the first entity text keyword, the tail entity represents the second entity text keyword that has a relation to the head entity, and the relation represents the association between the tail entity and the head entity. The head entity of the entity layer is associated with the leaf categories of the category layer.

[0130] Step 103: Based on the entity text keywords of the defined sensitive information categories, obtain the corresponding images and their structured information from the dataset. Construct image triples using the obtained images, and build an image hosting layer using the image triples. Each image triple includes: a head entity, a relation, and a tail entity. The tail entity is the image itself, the head entity is either the head or tail entity of the entity layer, and the relation is the structured information of the image, representing the association between the head and tail entities. For example, based on entity text keywords, crawl the corresponding images and structured information from Wikipedia to construct image triples. Image triples are a basic representation of multimodal knowledge graphs. Each triple includes <subject, predicate, object>, or can be represented as <head entity, relation, tail entity>. Specific examples: <reform and opening up (head entity), goal (relation), common prosperity (tail entity)>. The relationship of the triple is based on this class, and the image class also contains the corresponding relationship hasImage. Example: <person entity, hasImage, image corresponding to the person entity>, where the person entity is located in the entity layer.

[0131] Step 104: Before constructing the entity layer using triples of entity text keywords, for the already constructed triples of entity text keywords, based on the similarity between entity text keywords, different entity text keywords in the head entity with similarity exceeding a set similarity threshold are represented by the same node identifier; different entity text keywords in the tail entity with similarity exceeding a set similarity threshold are represented by the same node identifier; and different relations with the same node identifier are represented by the same node identifier. Triples of entity text keywords containing sensitive information that do not belong to the set category, as well as triples of erroneous entity text keywords, are removed. The remaining triples of entity text keywords are used to construct the entity layer. Before constructing the image layer using image triples, for the already constructed triples of images, triples of images containing sensitive information that do not belong to the set category, as well as triples of erroneous images, are removed. The remaining triples of images are used to construct the image hosting layer. That is: based on short text similarity calculation, entity unification and relation unification are achieved, because entity and relation unification means that entities and relations must be unique in the knowledge graph. "Entity consistency" means that if a political figure has different names or titles, then there must be a common node ID in the graph to represent the unique political figure entity. "Relationship consistency" means that terms like "wife," "wife's wife," and "daughter-in-law" can all refer to a marital relationship, so there must be a common `relationid` in the graph. Both entity consistency and relationship consistency can be achieved through short text similarity filtering.

[0132] Finally, the triplet information was cleaned by manual annotation, removing triplets that did not meet the database entry criteria (such as non-political content) or incorrect triplet information (errors occurred during crawling).

[0133] Step 105: The constructed political multimodal knowledge graph structure comprises three parts: a political category layer, an entity layer, and an image hosting layer. The political category layer is constructed by domain experts, who also collect and organize the connections between entities and subcategories within the current affairs category layer. The entity layer mainly consists of entities and relationships; entity categories primarily include ontology structures such as people, events, systems, and organizations. The image hosting layer stores image modal information (i.e., images), including faces of current affairs figures, sensitive images, and sensitive symbols. Images in the image hosting layer serve as image-type entity nodes connected to the entity layer; the image hosting layer only stores images.

[0134] Step 2: Multi-modal knowledge graph entity embedding training

[0135] For multimodal knowledge graphs, the embedding representation vectors of nodes and edges are trained by combining text feature extraction networks, image feature extraction networks, and the structure of multimodal knowledge graphs. The entity embedding training process for multimodal knowledge graphs is attached. Figure 5 As shown.

[0136] Step 201: After constructing the entity layer using triples of each entity text keyword, iterate through the triples of each entity text keyword within the entity layer. This involves randomly iterating through the triple relationships of entity text keywords across multiple knowledge graphs.<head,relation,tail> information;

[0137] Step 202: For the image entity in the triplet of an image, the image information is used to represent it; for other entities, the entity information is represented by entity text plus descriptive information; the relationship is represented by text. That is: the head entity is represented by the first entity text keywords and the descriptive information of the entity; the tail entity is represented by the second entity text keywords and the descriptive information of the entity; and the relationship is represented by text.

[0138] Step 203: The entities and relations within the triples of entity text keywords are input into the text feature extraction network to extract text feature vector embeddings; the image entities are input into the image feature extraction network to obtain image feature vector embeddings; specifically: the head entity, tail entity, and relation in the triples of the entity text keywords are input into the pre-trained text feature extraction network, and the text feature vectors of the head entity, tail entity, and relation are extracted by the pre-trained text feature extraction network respectively, and the text feature vectors of the triples of the entity text keywords are output, and the text feature vectors of the triples of each entity text keyword are saved to the entity vector table.

[0139] Step 204: The text feature extraction network uses the Robert pre-trained model, and the image feature extraction network uses ImageNet. Both the text and image feature extraction networks are pre-trained.

[0140] Step 205: The model training loss function adopts transE loss and language model loss, wherein the transE loss is V head +V relation =V tail The language model loss mentioned above is the pre-trained language model loss.

[0141] Step 206: After model training, the vectors corresponding to entities are stored in the entity vector table, which can improve the prediction speed when predicting entity links.

[0142] Specifically, the pre-trained text feature extraction network is trained using the following method:

[0143] The head entities, relations, and tail entities of the training samples are input into the convolutional network of the general knowledge base transE and the Robert language model for iterative training until the loss function of the text feature extraction network reaches the model convergence condition, thus obtaining the pre-trained text feature extraction network. The specific steps for iterative model training include:

[0144] The head entities, relations, and tail entities of the training samples are input into the convolutional network of the general knowledge base transE to obtain their respective low-dimensional vectors of text features.

[0145] Input the low-dimensional vectors of text features of the head entity, relation, and tail entity into the Robert language model. Use the first preset probability to hide the characters in the low-dimensional vectors of text features of the head entity, relation, and tail entity, and use the second preset probability to randomly replace the characters in the low-dimensional vectors of text features of the head entity, relation, and tail entity. Output the processed low-dimensional vectors of text features of the head entity, relation, and tail entity.

[0146] The loss function of the text feature extraction network includes the loss function of the general knowledge base transE and the loss function of the Robert language model. The loss function of the general knowledge base transE represents the relationship between the sum of the low-dimensional text feature vectors of the head entity and the low-dimensional text feature vectors of the relation and the low-dimensional text feature vectors of the tail entity. The loss function of the Robert language model represents the cross-entropy of the low-dimensional text feature vectors of the head entity, relation and tail entity after processing.

[0147] Step 3 Entity Link Prediction

[0148] The entity link dataset is constructed, and the entity link flowchart is attached. Figure 6 As shown.

[0149] The information to be published is traversed through the trie of the sensitive information. Based on the entities in the trie, fragments of the entities to be published and their positions are extracted. The fragments of the entities are then paired with the corresponding entities in the trie to generate candidate entity pairs. See steps 301-302 for details.

[0150] Step 301: The input text is processed through a trie to extract mention location information. The trie is constructed from entity names and entity references. A mention generally refers to a linguistic fragment in natural language that expresses an entity. For example, in the sentence "A certain meeting affirmed and established a certain thing done by a certain person," both the words "a certain meeting" and "a certain person" can be considered mentions. A mention refers to the "certain meeting" in the knowledge graph, and a mention refers to the corresponding "person entity" in the knowledge graph.

[0151] Step 302, candidate entity pair generation. Based on the trie matching results, candidate entity pairs are generated.<mention,entities> .

[0152] Step 303, entity matching. Because of candidate entity pairs...<mention,entities> In this context, there are numerous erroneous link results or entity ambiguities. Therefore, an entity matching model is constructed to improve the accuracy of entity linking. The structure of the entity matching model is shown in the attached figure. Figure 7 As shown. The input to the entity matching model consists of Input text, mention location information, and Input image (optional; when the input information includes an image, the image information can be used to enhance the recognition effect of the entity matching model).

[0153] Step 303-1: If the information to be published does not contain images, a pre-trained text feature extraction network and a pre-trained text matching network are used as the entity matching model. Specifically, this includes: inputting the information to be published into the pre-trained text feature extraction network to extract the low-dimensional vector of each character in the information; inputting the low-dimensional vectors of the characters at the corresponding positions of the entity segments into the pre-trained text feature extraction network to output the expression vectors corresponding to the entity segments; extracting entity vectors corresponding to the entity segments from the entity vector table; and inputting the expression vectors of the entity segments and the corresponding entity vectors into the pre-trained text matching network for matching prediction to obtain the first text matching value of the information to be published. For example: inputting the input text into the pre-trained text feature extraction network in step two to complete token embedding extraction. Selecting the word vector at the mention position and inputting it into a convolutional network to fuse mention information, constructing the mention vector V. M The entities entity extracts entity vector information V based on the entity vector table constructed in step two. E Construct Text MatchNet (text matching network) to match V... M and V E The text is input into a text matching network for matching prediction, and a text matching score P is obtained. text .

[0154] Step 303-2: When the input mention has an accompanying image, the entity matching result is enhanced by using the accompanying image.

[0155] The text information in the information to be published is input into a pre-trained text feature extraction network to extract the low-dimensional vector of each character in the text information; the low-dimensional vectors of the characters at the corresponding positions of the segments representing entities are input into the pre-trained text feature extraction network to output the expression vectors corresponding to the segments representing entities; entity vectors corresponding to the segments representing entities are extracted from the entity vector table; the expression vectors of the segments representing entities and the entity vectors corresponding to the segments representing entities are input into a pre-trained text matching network for matching prediction to obtain the second text matching value of the information to be published.

[0156] The information to be published is input into a pre-trained image feature extraction network to extract image features within the information to be published, resulting in a first image feature vector. The entities corresponding to the images contained within the information to be published are then input into the pre-trained image feature extraction network to extract the head entity vector and the relationship vector between the head and tail entities. The vector composed of the head entity vector and the relationship vector is used as the second image feature vector for the corresponding entity in the image. The first and second image feature vectors are then input into a pre-trained image matching network to obtain the image matching value for the information to be published. The second text matching value and the image matching value are used as the matching value between the fragment representing the entity and the corresponding entity in the candidate entity pair. For example: Inputting the accompanying image into the pre-trained image feature extraction network in step two extracts the first image feature vector I. M Using V E +V hasImage =I E The image features representing entities are used, where hasImage represents the relationship between entities and images in the graph database, resulting in the second image feature vector. After constructing the Image Match Net, the first image feature vector I is... M Second image feature vector (entity feature I) E The image is input into an image matching network for matching prediction, and the image matching score P is obtained. image .

[0157] Step 303-3: To address the issue of multi-modal training data collection, this invention decouples the matching networks of different modalities (i.e., when the mention contains an image, the text matching network and the image matching network operate independently), and uses a strategy fusion approach to fuse the entity matching models. If the information to be published does not contain an image, the candidate entity pair is determined to be the target candidate entity pair if the first text matching value is greater than the first text configuration threshold. If the information to be published contains an image, the candidate entity pair is determined to be the target candidate entity pair if the second text matching value is greater than the second text configuration threshold and the image matching value is greater than the image configuration threshold; wherein, the first text configuration threshold is greater than the second text configuration threshold. The matching value refers to the probability of entity linking; the higher the probability value, the more similar the mention and the entity are. A recognition result of True indicates that the mention and the entity are matched. Image knowledge is used to assist text-based entity linking and improve the accuracy of entity linking.

[0158] Steps 303-4: The Text Match Net and Image Match Net described in this invention can be trained on single-modal data respectively.

[0159] Step 304: Based on the multimodal knowledge graph, find the corresponding category of the entity matched in the target candidate entity pair and predict the recognition result.

[0160] The matched entity and mention location are used as prompt information and pushed to downstream reviewers to improve review efficiency.

[0161] The beneficial effects achieved by this invention are as follows:

[0162] By constructing a multimodal knowledge graph using expert knowledge and establishing connections between category and entity layers, the classification and identification problem is mapped to an entity linking problem, improving the interpretability of the model. Furthermore, by identifying scattered entities associated with categories, the data sparsity problem of sensitive information is effectively alleviated. When sensitive information is identified and pushed to downstream manual review, the reasons for the identification can be provided, greatly improving review efficiency. This invention avoids the technical problem of existing technologies where "model training effects rely entirely on corpus features, making it difficult to effectively improve the identification and coverage of politically sensitive information with sparse sample distribution and high harm." This invention fully utilizes multimodal information through multimodal knowledge graph pre-training and entity matching, improving the identification capability of sensitive information through entity links.

[0163] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0164] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0165] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0166] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0167] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0168] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0169] The steps of the methods or algorithms described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0170] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0171] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying sensitive information, characterized in that, include: Obtain the information to be published, extract fragments of express entities from the information to be published based on the trie of sensitive information, and generate candidate entity pairs by combining the fragments of express entities with the corresponding entities in the trie. The trie includes entities and their pronouns, and the entity refers to entity text keywords representing sensitive information or the corresponding image of the entity text keywords. The candidate entity pair input entity matching model performs matching calculations on the fragments of the entity expressed in the candidate entity pair and the corresponding entities, and outputs the matching values ​​of the fragments of the entity expressed in the candidate entity pair and the corresponding entities. Candidate entity pairs whose matching values ​​meet the preset configuration threshold are taken as target candidate entity pairs. Based on the pre-constructed multimodal knowledge graph of sensitive information, the categories of entities corresponding to the fragments of the expressed entities in the target candidate entity pairs are obtained. Based on the category of the entity, the identification result of the information to be published is determined, and the information to be published is recalled; The multimodal knowledge graph of the sensitive information includes a category layer, an entity layer, and an image hosting layer, wherein the entity layer is associated with the category layer, and the image hosting layer is associated with the entity layer; The multimodal knowledge graph of the sensitive information is constructed through the following methods: Collect category keywords for sensitive information in a defined category, create a category tree structure based on the category keywords to obtain a category layer, the tree structure includes leaf categories, and the leaf categories are used to directly connect to the entity layer; Identify the entity category to which words in the text containing the defined sensitive information belong, obtain entity text keywords for each entity category, construct triples of entity text keywords using entity text keywords, and construct an entity layer using the triples of entity text keywords; wherein, the triples of entity text keywords include: head entity, relation, and tail entity, the head entity represents a first entity text keyword, the tail entity represents a second entity text keyword that has a relation to the head entity, and the relation represents the association relationship between the tail entity and the head entity; Associate the head entity of the entity layer with the leaf category of the category layer; Based on entity text keywords of sensitive information in a defined category, images and their structured information corresponding to the entity text keywords are obtained from the dataset. Triples of the obtained images are constructed, and an image bed layer is constructed using the triples of each image. The triples of each image include: head entity, relation, and tail entity. The tail entity is the image itself, the head entity is either the head entity or the tail entity of the entity layer, and the relation is the structured information of the image. The structured information of the image is used to represent the association between the head entity and the tail entity.

2. The sensitive information identification method according to claim 1, characterized in that, The sensitive information identification method further includes: Based on the entities in the entity layer or image hosting layer of the multimodal knowledge graph of the sensitive information, a trie of the sensitive information is constructed. The trie includes entities and their pronouns. The entities include the head entity and tail entity in the triple of entity text keywords, and the tail entity in the triple of images.

3. The sensitive information identification method according to claim 1, characterized in that, Before constructing the entity layer using triples of each entity's text keywords, the following is also included: For the constructed triples of entity text keywords, based on the similarity between entity text keywords, different entity text keywords in the head entity whose similarity exceeds a set similarity threshold are represented by the same node identifier, different entity text keywords in the tail entity whose similarity exceeds a set similarity threshold are represented by the same node identifier, and different relations with the same node identifier in the relation are represented by the same node identifier; triples of entity text keywords that do not belong to the set category of sensitive information and triples of erroneous entity text keywords are removed, and the remaining triples of entity text keywords are used to construct the entity layer; Before constructing the image layer using image triples, the method further includes: For each constructed image triplet, remove the triplets of images containing sensitive information that does not belong to the defined category, as well as the triplets of erroneous images. The remaining image triplets are used to construct the image hosting layer.

4. The sensitive information identification method according to claim 1, characterized in that, Also includes: After constructing the entity layer using triples of entity text keywords, the triples of entity text keywords within the entity layer are traversed. The head entity is represented by the first entity text keyword and the descriptive information of the entity, the tail entity is represented by the second entity text keyword and the descriptive information of the entity, and the relationship is represented by text. The head entity, tail entity, and relation in the triplet of the entity text keyword are sequentially input into a pre-trained text feature extraction network. The pre-trained text feature extraction network extracts the text feature vectors of the head entity, tail entity, and relation respectively, and outputs the text feature vector of the triplet of the entity text keyword. The text feature vector of the triplet of each entity text keyword is saved to the entity vector table. The pre-trained text feature extraction network is trained using the following method: The head entities, relations, and tail entities of the training samples are input into the convolutional network of the general knowledge base transE and the Robert language model for iterative training until the loss function of the text feature extraction network reaches the model convergence condition, thus obtaining the pre-trained text feature extraction network. The specific steps for iterative model training include: The head entities, relations, and tail entities of the training samples are input into the convolutional network of the general knowledge base transE to obtain their respective low-dimensional vectors of text features. Input the low-dimensional vectors of text features of the head entity, relation, and tail entity into the Robert language model. Use the first preset probability to hide the characters in the low-dimensional vectors of text features of the head entity, relation, and tail entity. Use the second preset probability to randomly replace the characters in the low-dimensional vectors of text features of the head entity, relation, and tail entity. Output the processed low-dimensional vectors of text features of the head entity, relation, and tail entity. The loss function of the text feature extraction network includes the loss function of the general knowledge base transE and the loss function of the Robert language model. The loss function of the general knowledge base transE represents the relationship between the sum of the low-dimensional text feature vectors of the head entity and the low-dimensional text feature vectors of the relation and the low-dimensional text feature vectors of the tail entity. The loss function of the Robert language model represents the cross-entropy of the low-dimensional text feature vectors of the head entity, relation and tail entity after processing.

5. The sensitive information identification method according to claim 4, characterized in that, The trie based on sensitive information extracts fragments of expressive entities from the information to be published, and generates candidate entity pairs by combining these fragments with corresponding entities in the trie, including: The information to be published is traversed through the trie of the sensitive information. Based on the entities in the trie, the fragments of the entities to be published and their positions are extracted. The fragments of the entities are then paired with the corresponding entities in the trie to generate candidate entity pairs. The step of inputting the candidate entity pair into the entity matching model to perform matching calculations on the fragments of the entity expressed in the candidate entity pair with the corresponding entities, and outputting the matching values ​​of the fragments of the entity expressed in the candidate entity pair with the corresponding entities, includes: If the information to be published does not contain images, a pre-trained text feature extraction network and a pre-trained text matching network are used as the entity matching model, specifically including: The information to be published is input into a pre-trained text feature extraction network to extract the low-dimensional vector of each character in the information to be published; the low-dimensional vectors of the characters at the corresponding positions of the segments representing entities are input into the pre-trained text feature extraction network to output the expression vectors corresponding to the segments representing entities. Extract entity vectors corresponding to the segments representing the entities from the entity vector table; The expression vector of the entity segment and the corresponding entity vector are input into a pre-trained text matching network for matching prediction to obtain the first text matching value of the information to be published. The first text matching value is used as the matching value between the fragment representing the entity and the corresponding entity in the candidate entity pair; If the information to be published contains images, a decoupled pre-trained text feature extraction network and a pre-trained image feature extraction network, as well as a pre-trained text matching network and a pre-trained image matching network, are used as the entity matching model. Specifically, this includes: The text information in the information to be published is input into a pre-trained text feature extraction network to extract the low-dimensional vector of each character in the text information; the low-dimensional vectors of the characters at the corresponding positions of the segments representing entities are input into the pre-trained text feature extraction network to output the expression vectors corresponding to the segments representing entities. Extract entity vectors corresponding to the segments representing the entities from the entity vector table; The expression vector of the entity segment and the entity vector corresponding to the entity segment are input into a pre-trained text matching network for matching prediction to obtain the second text matching value of the information to be published. as well as, The information to be published is input into a pre-trained image feature extraction network to extract image features within the information to be published, thus obtaining the first image feature vector within the information to be published. The entities corresponding to the images contained in the information to be published are input into a pre-trained image feature extraction network to extract the head entity vector and the relationship vector between the head entity and the tail entity. The vector composed of the head entity vector and the relationship vector is used as the second image feature vector of the corresponding entity of the image. The first image feature vector and the second image feature vector are input into a pre-trained image matching network to obtain the image matching value of the information to be published. The second text matching value and image matching value are used as the matching values ​​between the fragment representing the entity and the corresponding entity in the candidate entity pair.

6. The sensitive information identification method according to claim 5, characterized in that, The method for determining the target candidate entity pair includes: If the information to be published does not contain an image, and the first text matching value is greater than the first text configuration threshold, the candidate entity pair is determined to be the target candidate entity pair. If the information to be published contains an image, and the second text matching value is greater than the second text configuration threshold and the image matching value is greater than the image configuration threshold, the candidate entity pair is determined to be the target candidate entity pair; wherein, the first text configuration threshold is greater than the second text configuration threshold.

7. The sensitive information identification method according to claim 5, characterized in that, The multimodal knowledge graph based on pre-constructed sensitive information obtains the categories of entities corresponding to fragments of expressed entities in the target candidate entity pairs, including: The graph database is used to find the category of the corresponding entity in the target candidate pair in the multimodal knowledge graph of the sensitive information; The determination of the identification result of the information to be published based on the category of the entity includes: The category of the entity, the entity itself, and the position of the fragment representing the entity are used as the identification results of the information to be published.

8. A sensitive information identification system, characterized in that, include: The traversal unit is used to obtain the information to be published, extract fragments of the expressive entity from the information to be published based on the trie of sensitive information, and generate candidate entity pairs by combining the fragments of the expressive entity with the corresponding entities in the trie. The trie includes entities and their pronouns. The entity refers to the entity text keyword representing sensitive information or the corresponding image of the entity text keyword. An entity matching model is used to perform matching calculations between fragments representing entities in the input candidate entity pairs and corresponding entities, and outputs the matching values ​​between fragments representing entities in the candidate entity pairs and corresponding entities. The category determination unit is used to take the candidate entity pairs whose matching values ​​meet the preset configuration threshold as target candidate entity pairs, and obtain the category of the entity corresponding to the fragment of the expressed entity in the target candidate entity pair based on the pre-constructed multimodal knowledge graph of sensitive information. The recall unit is used to determine the identification result of the information to be published based on the category of the entity, and recall the information to be published. The multimodal knowledge graph of the sensitive information includes a category layer, an entity layer, and an image hosting layer, wherein the entity layer is associated with the category layer, and the image hosting layer is associated with the entity layer; A multimodal knowledge graph of sensitive information is constructed using category-level, entity-level, and image hosting-level building units, where: The category layer construction unit is used to collect category keywords for sensitive information of a set category, and create a tree structure of categories based on the category keywords to obtain a category layer. The tree structure includes leaf categories, which are used to be directly connected to the entity layer. The entity layer construction unit is used to identify the entity category to which words in the text containing the defined sensitive information belong, obtain entity text keywords for each entity category, construct triples of entity text keywords using the entity text keywords, and construct the entity layer through the triples of each entity text keyword; wherein, the triples of the entity text keywords include: head entity, relation, and tail entity, the head entity represents a first entity text keyword, the tail entity represents a second entity text keyword that has an association relationship with the head entity, the relation represents the association relationship between the tail entity and the head entity; and associate the head entity of the entity layer with the leaf category of the category layer; The image hosting layer construction unit is used to obtain the corresponding images and their structured information from the dataset based on the entity text keywords of the sensitive information of the set category, construct triples of the images using the obtained images, and construct the image hosting layer through the triples of each image; wherein, the triples of the images include: head entity, relation, and tail entity, the tail entity is the image, the head entity is the head entity or tail entity of the entity layer, the relation is the structured information of the image, and the structured information of the image is used to represent the association relationship between the head entity and the tail entity.

Images