Sensitive word filtering method and device based on multiple modes
By analyzing image and text data using multimodal causal knowledge graphs, this study solves the problems of high false positive rates and cross-modal attacks in existing sensitive word filtering technologies, achieving high-precision sensitive word filtering and security defense.
Patent Information
- Application Number
- CN202511078505.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-01
- Publication Date
- 2025-11-11
AI Technical Summary
Existing sensitive word filtering technologies cannot effectively distinguish whether words lead to harmful intent, resulting in a high false alarm rate. Furthermore, they cannot cope with cross-modal attacks and have security blind spots.
By acquiring image and text data, performing feature extraction, and mapping it to a multimodal causal knowledge graph, the causal knowledge graph is used for sensitive word filtering to distinguish between harmful intentions and harmless discussions.
It achieves high-precision, low-false-positive sensitive word filtering, effectively resisting cross-modal attacks and improving security.
Smart Images

Figure CN120929593A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of sensitive word filtering technology, and in particular to a sensitive word filtering method and apparatus based on multimodality. Background Technology
[0002] Security defenses for large-scale language models are the core bottleneck for their widespread application. Existing Prompt security detection technologies, whether based on keyword matching, traditional machine learning classification, or a single deep learning model, face increasingly severe and insurmountable challenges: relevance does not equal causation, leading to high false positive rates; existing models primarily learn the correlation between text features and "harmful" labels.
[0003] Current technologies lack causal inference capabilities, failing to distinguish whether a word indicates malicious intent or merely accompanies harmless discussion. This results in an extremely high false positive rate, severely impacting the user experience for legitimate users. The limitations of unimodal defenses also make them ineffective against cross-modal attacks: with the development of multimodal large-scale models, attack methods have expanded from plain text to combinations of text and images. For example, an attacker might upload a harmless image (such as a cat) but describe it in a cryptic way in the prompt, combining the two to form a malicious instruction. Existing plain text defense methods are completely powerless against such cross-modal attacks, creating a significant security blind spot.
[0004] Based on this, the present invention proposes a multimodal sensitive word filtering method and apparatus to solve the problem of how to accurately filter sensitive words in text. Summary of the Invention
[0005] To address the problem of accurately filtering sensitive words in text, embodiments of the present invention provide a multimodal sensitive word filtering method and apparatus.
[0006] In a first aspect, embodiments of the present invention provide a multimodal sensitive word filtering method, the method comprising:
[0007] Acquire the data to be detected; wherein the data to be detected includes at least one image data to be detected and one text data to be detected;
[0008] The data to be detected is subjected to feature extraction to obtain image features and text features;
[0009] The image features and text features are mapped onto a multimodal causal knowledge graph to obtain a causal knowledge graph to be detected; wherein, the causal knowledge graph to be detected is used to characterize the sensitive relationship between the image and text data to be detected in the data to be detected;
[0010] Based on the causal knowledge graph to be detected, sensitive words are filtered on the data to be detected.
[0011] Secondly, embodiments of the present invention provide a multimodal sensitive word filtering device, comprising:
[0012] An acquisition module is used to acquire data to be detected; wherein, the data to be detected includes at least one image data to be detected and one text data to be detected;
[0013] The first data processing module is used to extract features from the data to be detected to obtain image features and text features;
[0014] The second data processing module is used to map the image features and text features to a multimodal causal knowledge graph to obtain a causal knowledge graph to be detected; wherein, the causal knowledge graph to be detected is used to characterize the sensitive relationship between the image to be detected and the text to be detected in the data to be detected;
[0015] The third data processing module is used to filter sensitive words in the data to be detected based on the causal knowledge graph to be detected.
[0016] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method described in any embodiment of the present invention.
[0017] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in any embodiment of the present invention.
[0018] This invention provides a multimodal sensitive word filtering method and apparatus. First, it acquires the data to be detected, which must include at least one set of image data and corresponding text data. This data can originate from user-uploaded image-text combinations, multimedia information published on social media platforms, and other scenarios. After acquisition, feature extraction is performed on both types of data: for the text data, deep semantic parsing is performed using a pre-trained language model (such as BERT) to extract text features including context, sentiment, and the distribution of potential sensitive words; for the image data, image features such as body contours, scene information, and visual semantic vectors are extracted using a visual Transformer model (such as ViT). Subsequently, the extracted image features and text features are jointly mapped to a multimodal causal knowledge graph using cross-modal embedding technology. This knowledge graph comprises entity nodes (such as sensitive words and image objects), semantic association edges (such as the matching degree between text descriptions and image content), and causal paths (such as the causal relationship between sensitive words and harmful intent). The resulting causal knowledge graph can accurately represent the association logic between the image and text data at the sensitive information level, including the occurrence scenarios of sensitive words in the text and the complementary or conflicting relationship between image content and text semantics. Finally, sensitive word filtering is performed based on the causal knowledge graph. By traversing the causal paths and association weights in the graph, it is possible to effectively distinguish whether sensitive words in the text are core factors constituting harmful intent or accompanying words in a harmless context. For example, when the word "protest" appears in news report text and matches the news scene features of the image, it is judged as a normal expression; if it is combined with obscure instruction text and unrelated images, and the graph shows that it has a direct causal relationship with harmful intent, the filtering mechanism is triggered. Thus, through the collaborative analysis of multimodal information, accurate filtering of sensitive words in the text is achieved. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 A flowchart of a multimodal-based sensitive word filtering method according to one embodiment is shown;
[0021] Figure 2 This is a hardware architecture diagram of an electronic device provided in an embodiment of the present invention;
[0022] Figure 3A structural diagram of a multimodal-based sensitive word filtering device according to one embodiment is shown. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0024] Please refer to Figure 1 This invention provides a multimodal sensitive word filtering method, the method comprising:
[0025] Step 100: Obtain the data to be detected; wherein, the data to be detected includes at least one image data to be detected and one text data to be detected;
[0026] Step 102: Extract features from the data to be detected to obtain image features and text features;
[0027] Step 104: Map image features and text features to a multimodal causal knowledge graph to obtain the causal knowledge graph to be detected; wherein, the causal knowledge graph to be detected is used to characterize the sensitive relationship between the image to be detected and the text to be detected in the data to be detected;
[0028] Step 106: Based on the causal knowledge graph to be detected, perform sensitive word filtering on the data to be detected.
[0029] In this embodiment, the acquisition of the data to be detected is performed first. The acquired data must include at least one set of image data and corresponding text data to be detected. This data can originate from user-uploaded image and text combinations, multimedia information published on social media platforms, and other scenarios. After acquisition, feature extraction is performed on the two types of data: for the text data to be detected, deep semantic parsing is performed using a pre-trained language model (such as BERT) to extract text features including context, sentiment, and distribution of potential sensitive words; for the image data to be detected, image features such as body contours, scene information, and visual semantic vectors are extracted using a visual Transformer model (such as ViT). Subsequently, the extracted image features and text features are jointly mapped to a multimodal causal knowledge graph using cross-modal embedding technology. This knowledge graph comprises entity nodes (such as sensitive words and image objects), semantic association edges (such as the matching degree between text descriptions and image content), and causal paths (such as the causal relationship between sensitive words and harmful intent). The resulting causal knowledge graph can accurately represent the association logic between the image and text data at the sensitive information level, including the occurrence scenarios of sensitive words in the text and the complementary or conflicting relationship between image content and text semantics. Finally, sensitive word filtering is performed based on the causal knowledge graph. By traversing the causal paths and association weights in the graph, it is possible to effectively distinguish whether sensitive words in the text are core factors constituting harmful intent or accompanying words in a harmless context. For example, when the word "protest" appears in news report text and matches the news scene features of the image, it is judged as a normal expression; if it is combined with obscure instruction text and unrelated images, and the graph shows that it has a direct causal relationship with harmful intent, the filtering mechanism is triggered. Thus, through the collaborative analysis of multimodal information, accurate filtering of sensitive words in the text is achieved.
[0030] In one embodiment of the present invention, the multimodal causal knowledge graph includes entity nodes, concept nodes, intent nodes, sentiment nodes and modal nodes that are interconnected through preset edge relationships;
[0031] Among them, edge relationships include causal relationship edges, association relationship edges, and attribute relationship edges.
[0032] In this embodiment, the core architecture of the multimodal causal knowledge graph consists of entity nodes, concept nodes, intent nodes, sentiment nodes, and modal nodes, which are organically linked through pre-defined edge relationships. Entity nodes cover specific referents, such as real-world entities like names ("Zhang San") and place names ("a square"); concept nodes correspond to abstract categories, including sensitive concepts such as "violence" and "discrimination" as well as neutral concepts such as "public affairs"; intent nodes focus on user behavior goals, subdivided into benign intents such as "seeking information" and "normal communication" and malicious intents such as "inducing the generation of harmful content" and "evading detection"; sentiment nodes capture the emotional tendencies in text and images, such as negative emotions like "anger" and "incitement" as well as neutral emotions like "objectivity" and "neutrality"; modal nodes mark the type of information carrier, such as "text paragraph" and "static image". Edge relationships, serving as connecting links between nodes, include: causal edges (Causes) characterizing causal transmission between nodes (e.g., "inducing generation" intention → "violence" concept); associated edges (AssociatedWith) reflecting the co-occurrence association of nodes (e.g., "anger" emotion and "discrimination" concept); and attribute edges (HasProperty) describing the inherent characteristics of nodes (e.g., "text" modality → "contains sensitive words" attribute). Through the collaboration of multiple types of nodes and edges, a knowledge network covering the causal logic of multimodal information is constructed.
[0033] In one embodiment of the present invention, a multimodal causal knowledge graph is constructed through the following steps:
[0034] Acquire historical training data; historical training data includes historical text data and historical image data;
[0035] Input historical training data into a pre-set large language model to obtain historical text node data, historical image node data, and edge relationships between historical text data and historical image data;
[0036] Based on historical text node data, historical image node data, and edge relationships between historical text data and historical image data, an initial causal knowledge graph is determined.
[0037] Based on the initial causal knowledge graph, a multimodal causal knowledge graph is determined.
[0038] In this embodiment, the first step is to acquire historical training data. This data covers historical text data (such as public documents and user interaction records) and historical image data (such as news images and social media platform images) from diverse scenarios, and includes manually labeled sensitive / normal tags and modal association information (such as "text describing image content"). Subsequently, the preprocessed historical training data is input into a pre-defined large language model (such as a multimodal LLM combined with a visual encoder). The model automatically extracts node data such as entities, concepts, and intentions from historical text (i.e., historical text node data) and node data such as visual entities and scene concepts from historical images (i.e., historical image node data) through deep semantic parsing and cross-modal association learning. Simultaneously, it generates edge relationships between the two types of data (such as a causal relationship edge between the text "protest" and the image "gathering scene," or an association relationship edge between text sentiment and image color). Based on the extracted historical text node data, historical image node data, and corresponding edge relationships, a graph fusion algorithm is used to integrate duplicate nodes and correct conflicting relationships, forming an initial causal knowledge graph containing multiple types of nodes such as entities, concepts, and intentions, as well as causal and association edge relationships. Finally, the initial graph was iteratively optimized: by combining the corrected data annotated by domain experts, the node definitions and edge relationship weights were adjusted (such as strengthening the causal strength of the "induced generation" intention and the "violent" concept), and new nodes and edge relationships that frequently appeared were added, and finally a multimodal causal knowledge graph with rigorous causal logic covering multimodal scenarios was determined.
[0039] In one embodiment of the present invention, determining a multimodal causal knowledge graph based on an initial causal knowledge graph includes:
[0040] A multimodal causal knowledge graph is obtained by pre-training the initial causal knowledge graph using a graph neural network and pre-setting a path weight for the edge relationship between every two nodes in the initial causal knowledge graph.
[0041] Among them, path weight is used to measure the sensitivity between corresponding nodes.
[0042] In this embodiment, the process of determining a multimodal causal knowledge graph based on an initial causal knowledge graph is as follows: A graph neural network (such as GNN and its variants) is introduced to pre-train the initial causal knowledge graph. Iterative learning optimizes the representation capabilities of node features and edge relationships, enabling the model to deeply capture the potential associations between different types of nodes, such as entities, concepts, and intentions. Simultaneously, for each pair of nodes in the initial causal knowledge graph, corresponding path weights are preset for edge relationships (including causal, associative, and attribute relationships). These weights are dynamically adjusted by analyzing sensitive case samples in historical multimodal data and combining them with a supervised learning mechanism to accurately measure the sensitivity of associations between nodes. For example, the path weight of the causal relationship edge between the intention node "induced generation" and the concept node "violence" will be significantly higher than that of the associative relationship edge between the entity node "place name" and the modal node "text," thereby strengthening the ability to identify highly sensitive association paths. Ultimately, a multimodal causal knowledge graph that combines causal logic and sensitivity level quantification is formed.
[0043] In one embodiment of the present invention, the causal knowledge graph to be detected includes at least two nodes and one edge relationship;
[0044] Based on the causal knowledge graph to be detected, sensitive word filtering is performed on the data to be detected, including:
[0045] The path weights of the edge relationships between every two nodes in the causal knowledge graph to be tested are statistically analyzed to obtain one or more path scores.
[0046] The scores of one or more paths are summed to obtain a comprehensive score;
[0047] Based on the comprehensive score, sensitive words are filtered out from the data to be detected.
[0048] In this embodiment, the causal knowledge graph to be detected consists of at least two nodes (such as "sensitive word entity" in text and "scene concept" in image) and edge relationships connecting the nodes (such as causal relationship edges or association relationship edges). The process of sensitive word filtering based on this graph is as follows: First, the path weights of the edge relationships corresponding to each pair of nodes in the graph are calculated. These weights are derived from the preset parameters of the multimodal causal knowledge graph and directly reflect the sensitivity level of the node association. Then, the weight values of all related paths are summed to obtain a comprehensive score that measures the overall sensitivity of the data to be detected. For example, if the "protest" entity in text and the "conflict scene" concept in image are connected by a causal edge, and the weights of the two association paths are 0.6 and 0.3 respectively, the comprehensive score is 0.9. Finally, a preset threshold (such as 0.7) is used to determine: if the comprehensive score exceeds the threshold, it is determined that there are sensitive words that need to be filtered; otherwise, it is considered normal content, thereby achieving accurate screening of sensitive information.
[0049] In one embodiment of the present invention, sensitive word filtering is performed on the data to be detected based on a comprehensive score, including:
[0050] When the overall score is greater than the preset score, sensitive words are filtered out from the data to be tested.
[0051] In this embodiment, the specific method for filtering sensitive words in the data to be detected based on the comprehensive score is as follows: the preset score is a threshold value set by combining historical multimodal sensitive cases and security thresholds. When the comprehensive score of the data to be detected exceeds the preset score, it is determined that it has a high sensitivity risk. At this time, the sensitive word filtering mechanism is triggered to accurately intercept and process the sensitive words in the data.
[0052] In one embodiment of the present invention, after filtering the data to be detected for sensitive words when the overall score is greater than a preset score, the method further includes:
[0053] The causal knowledge graph to be tested is sent to the preset terminal.
[0054] In this embodiment, when the overall score exceeds the preset score and sensitive word filtering is completed, the causal knowledge graph to be detected is simultaneously sent to a preset terminal (such as a manual review platform). At this time, for prompts judged as "harmful," the system not only outputs the classification result but also generates an interpretable attribution analysis report based on the causal path with the highest score. For example, the report will clearly state: "This instruction is judged as harmful because 'chemical experimental equipment' and 'explosives' in the text have a strong correlation in the knowledge graph, and both point to the harmful concept of 'violence.'" Such reports provide manual reviewers with clear causal logic and decision-making basis, improving review efficiency and credibility.
[0055] In summary, this invention solves the problem of misjudgment caused by missing causal relationships, achieving high-precision, low-false-positive identification. Revolutionarily, this invention introduces a causal inference mechanism. By analyzing the causal path between input features and malicious intent, rather than superficial correlations, it can accurately distinguish between malicious commands and normal discussions, fundamentally solving the high false-positive rate problem caused by "correlation without causation" in existing technologies. It overcomes the blind spot of single-modal defense, possessing the ability to defend against cross-modal attacks: by constructing a multimodal causal knowledge graph, this invention can comprehensively analyze text and image information, identifying the complex semantics and intents generated by their combination. This enables this invention to effectively resist cross-modal attacks that use image-text combinations for camouflage, filling a significant security gap in existing pure text defense technologies.
[0056] like Figure 2 , Figure 3As shown, this embodiment of the invention provides a multimodal sensitive word filtering device. The device embodiment can be implemented through software, hardware, or a combination of both. From a hardware perspective, as... Figure 2 The diagram shown is a hardware architecture diagram of an electronic device containing a multimodal sensitive word filtering device according to an embodiment of the present invention. (Except for...) Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing packets. Taking software implementation as an example, such as... Figure 3 As shown, a device in a logical sense is formed by the CPU of the electronic device in which it is located reading the corresponding computer program from the non-volatile memory into the memory for execution.
[0057] like Figure 3 As shown in the figure, this embodiment provides a multimodal sensitive word filtering device, including:
[0058] The acquisition module 300 is used to acquire data to be detected; wherein, the data to be detected includes at least one image data to be detected and one text data to be detected;
[0059] The first data processing module 302 is used to extract features from the data to be detected to obtain image features and text features;
[0060] The second data processing module 304 is used to map the image features and text features to a multimodal causal knowledge graph to obtain a causal knowledge graph to be detected; wherein, the causal knowledge graph to be detected is used to characterize the sensitive relationship between the image to be detected and the text to be detected in the data to be detected;
[0061] The third data processing module 306 is used to filter sensitive words in the data to be detected based on the causal knowledge graph to be detected.
[0062] In one embodiment of the present invention, the multimodal causal knowledge graph includes entity nodes, concept nodes, intent nodes, sentiment nodes and modal nodes that are interconnected through preset edge relationships;
[0063] The edge relationships include causal relationship edges, association relationship edges, and attribute relationship edges.
[0064] In one embodiment of the present invention, the device further includes a fourth data processing module, the fourth data processing module being configured to perform the following operations:
[0065] Acquire historical training data; wherein, the historical training data includes historical text data and historical image data;
[0066] The historical training data is input into a preset large language model to obtain historical text node data, historical image node data, and edge relationships between historical text data and historical image data.
[0067] Based on the historical text node data, the historical image node data, and the edge relationships between the historical text data and the historical image data, an initial causal knowledge graph is determined;
[0068] Based on the initial causal knowledge graph, the multimodal causal knowledge graph is determined.
[0069] In one embodiment of the present invention, when the fourth data processing module 308 performs the step of determining the multimodal causal knowledge graph based on the initial causal knowledge graph, it performs the following operations:
[0070] The initial causal knowledge graph is pre-trained using a graph neural network, and a path weight is preset for the edge relationship between every two nodes in the initial causal knowledge graph to obtain the multimodal causal knowledge graph.
[0071] The path weight is used to measure the sensitivity between corresponding nodes.
[0072] In one embodiment of the present invention, the causal knowledge graph to be detected includes at least two nodes and one edge relationship; the third data processing module 306 is used to perform the following operations:
[0073] The path weights of the edge relationships between every two nodes in the causal knowledge graph to be detected are statistically analyzed to obtain one or more path scores.
[0074] The scores of one or more of the paths are summed to obtain a comprehensive score;
[0075] Based on the comprehensive score, sensitive words are filtered out from the data to be detected.
[0076] In one embodiment of the present invention, when the third data processing module 306 performs sensitive word filtering on the data to be detected based on the comprehensive score, it performs the following operations:
[0077] When the overall score is greater than the preset score, sensitive word filtering is performed on the data to be detected.
[0078] In one embodiment of the present invention, the device further includes a fifth data processing module, which is configured to perform the following operations:
[0079] The causal knowledge graph to be detected is sent to a preset terminal.
[0080] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on a multimodal sensitive word filtering device. In other embodiments of the present invention, a multimodal sensitive word filtering device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0081] The information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.
[0082] This invention also provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a multimodal sensitive word filtering method according to any embodiment of this invention.
[0083] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform a multimodal sensitive word filtering method according to any embodiment of this invention.
[0084] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.
[0085] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.
[0086] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.
[0087] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0088] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the functions of any of the embodiments described above.
[0089] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.
[0090] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.
[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multimodal sensitive word filtering method, characterized in that, The method includes: Acquire the data to be detected; wherein the data to be detected includes at least one image data to be detected and one text data to be detected; The data to be detected is subjected to feature extraction to obtain image features and text features; The image features and text features are mapped onto a multimodal causal knowledge graph to obtain a causal knowledge graph to be detected; wherein, the causal knowledge graph to be detected is used to characterize the sensitive relationship between the image and text data to be detected in the data to be detected; Based on the causal knowledge graph to be detected, sensitive words are filtered on the data to be detected.
2. The method according to claim 1, characterized in that, The multimodal causal knowledge graph includes entity nodes, concept nodes, intention nodes, emotion nodes, and modal nodes that are interconnected through preset edge relationships; The edge relationships include causal relationship edges, association relationship edges, and attribute relationship edges.
3. The method according to claim 2, characterized in that, The multimodal causal knowledge graph is constructed through the following steps: Acquire historical training data; wherein, the historical training data includes historical text data and historical image data; The historical training data is input into a preset large language model to obtain historical text node data, historical image node data, and edge relationships between historical text data and historical image data. Based on the historical text node data, the historical image node data, and the edge relationships between the historical text data and the historical image data, an initial causal knowledge graph is determined; Based on the initial causal knowledge graph, the multimodal causal knowledge graph is determined.
4. The method according to claim 3, characterized in that, The process of determining the multimodal causal knowledge graph based on the initial causal knowledge graph includes: The initial causal knowledge graph is pre-trained using a graph neural network, and a path weight is preset for the edge relationship between every two nodes in the initial causal knowledge graph to obtain the multimodal causal knowledge graph. The path weight is used to measure the sensitivity between corresponding nodes.
5. The method according to claim 4, characterized in that, The causal knowledge graph to be detected includes at least two nodes and one edge relationship; The process of filtering sensitive words in the data to be detected based on the causal knowledge graph to be detected includes: The path weights of the edge relationships between every two nodes in the causal knowledge graph to be detected are statistically analyzed to obtain one or more path scores. The scores of one or more of the paths are summed to obtain a comprehensive score; Based on the comprehensive score, sensitive words are filtered out from the data to be detected.
6. The method according to claim 5, characterized in that, Based on the comprehensive score, sensitive word filtering is performed on the data to be detected, including: When the overall score is greater than the preset score, sensitive word filtering is performed on the data to be detected.
7. The method according to claim 6, characterized in that, After filtering the data to be detected for sensitive words when the overall score is greater than the preset score, the method further includes: The causal knowledge graph to be detected is sent to a preset terminal.
8. A multimodal sensitive word filtering device, characterized in that, include: An acquisition module is used to acquire data to be detected; wherein, the data to be detected includes at least one image data to be detected and one text data to be detected; The first data processing module is used to extract features from the data to be detected to obtain image features and text features; The second data processing module is used to map the image features and text features to a multimodal causal knowledge graph to obtain a causal knowledge graph to be detected; wherein, the causal knowledge graph to be detected is used to characterize the sensitive relationship between the image to be detected and the text to be detected in the data to be detected; The third data processing module is used to filter sensitive words in the data to be detected based on the causal knowledge graph to be detected.
9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the method described in any one of claims 1-7.
Citation Information
Cited By
Multi-modal input content filtering method and system for educational scene
CN121478965A
A method and system for filtering multimodal input content in educational settings
CN121478965B
Prompt-enhanced online social network privacy perception method, system and device and medium
CN122263150A