Multimodal data sensitivity grading method and system based on semantic risk map diffusion perception

Abstract

The invention discloses a multi-modal data sensitivity grading method and system based on semantic risk map diffusion perception, and belongs to the technical field of artificial intelligence security and information content identification. Aiming at the problems of weak cross-modal linkage capability, insufficient context modeling, poor interpretability and the like of the existing multi-modal sensitivity identification method, the method realizes multi-modal fragment association by constructing a semantic risk unit, utilizes a semantic risk graph to model a cross-modal cooperative relationship, and adopts a thermal diffusion mechanism to simulate a risk propagation process, so that the method has the advantages of high sensitivity and high reliability. And finally, generating a structured interpretation path through the sensitive tag atlas. According to the method, high-precision multi-mode sensitive information identification and traceable grading judgment are realized, and the method is suitable for automatic compliance review of AIGC generation content.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence security and information content recognition technology, and in particular relates to a multimodal data sensitivity grading method and system based on semantic risk graph diffusion perception. Background Art

[0002] Current data sensitivity identification methods are primarily based on static rules or single-modality classifiers, such as sensitive keyword matching, image detection models, and speech recognition systems. While these methods have demonstrated some effectiveness in specific scenarios, they suffer from significant limitations in practical applications. Existing technologies lack cross-modal linkage perception, making it difficult to effectively identify semantic coupling relationships between text and images, or between video and audio. For example, it is impossible to associate an ID photo with an ID number in text. Furthermore, existing methods ignore structural dependencies between contexts, making it difficult to model "combined sensitivity" or "implicit triggering" scenarios. For example, sensitive information distributed across different modalities may appear low-risk individually, but when combined, it can become high-risk. Furthermore, traditional methods fail to output an interpretable sensitivity determination path, hindering compliance audits and risk backtracking, particularly in areas requiring rigorous compliance review. The root cause of these problems lies in the fact that traditional methods, when dealing with sensitive data with complex structures, rich modalities, and strong semantic linkages generated by AIGC, cannot effectively capture cross-modal semantic connections, lack the ability to model contextual risk propagation, and offer a transparent explanation of the determination process. These problems seriously restrict the accuracy and practicality of multimodal sensitive data identification. There is an urgent need for a sensitivity detection technology that has cross-modal modeling capabilities, supports context diffusion perception, and can output graded results and label path explanations. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention proposes a multimodal data sensitivity grading method and system based on semantic risk graph diffusion perception to solve the problems existing in the above-mentioned existing technologies.

[0004] In a first aspect, to achieve the above-mentioned objectives, the present invention provides a multimodal data sensitivity grading method based on semantic risk graph diffusion perception, comprising the following steps:

[0005] Perform modality recognition and structural processing on the input multimodal data, screen cross-modal segment combinations based on semantic coupling, and construct semantic risk units;

[0006] Extract the semantic embedding representation of each modality segment in the semantic risk unit, map it to the shared semantic space and perform weighted aggregation to generate a fusion representation vector;

[0007] Taking semantic risk units as nodes, a semantic risk graph is constructed based on semantic similarity, modality co-occurrence frequency, entity consistency and contextual relationship;

[0008] Initialize the risk pressure value for the nodes of the semantic risk graph, simulate the risk propagation based on the graph structure using the heat diffusion mechanism, and obtain the steady-state risk value of the node;

[0009] The steady-state risk values ​​of all nodes are integrated to construct a sample-level risk vector, which is then input into the scoring function to output the sensitivity level prediction result.

[0010] A sensitive label graph is introduced and jointly modeled with the semantic risk graph. A label path chain is generated through semantic path matching, and structured explanation results are output.

[0011] Optionally, the process of constructing the semantic risk unit includes:

[0012] Align multimodal data along a unified time axis to generate a triple representation containing modality type, timestamp, and embedded features;

[0013] Calculate the semantic similarity, temporal error and contextual consistency indicators of any two modal segments;

[0014] Based on a preset threshold, segment pairs that meet the semantic coupling condition are screened to form a candidate combination set;

[0015] The risk relevance score of candidate combinations is calculated based on semantic fusion strength, entity consistency and sensitive label directionality, and combinations with scores exceeding the threshold are screened to construct semantic risk units.

[0016] Optionally, the process of generating the fused representation vector includes:

[0017] Call the pre-trained model adapted to the modality type to extract the semantic feature representation of each modality segment;

[0018] Mapping features to a unified semantic space through a modality-specific projection matrix;

[0019] A weighted fusion strategy is adopted for the multimodal embedding representations within the same semantic risk unit to generate a cross-modal semantic fusion vector.

[0020] Optionally, the process of constructing the semantic risk map includes:

[0021] Map the semantic risk units to nodes of the graph, and use their fused representation vectors as node attributes;

[0022] Calculate the edge weights between nodes based on semantic similarity, co-referencing entity overlap, temporal relationship, sensitive label consistency, and modal co-occurrence frequency;

[0023] Generate an adjacency matrix of multi-relation expressions to form a semantic risk graph.

[0024] Optionally, the process of the heat diffusion mechanism includes:

[0025] Initialize the node risk pressure value according to label confidence, modal coverage completeness, semantic ambiguity and entity recognition confidence;

[0026] Iteratively update the pressure value based on the relationship between adjacent nodes until the average change is lower than the convergence threshold;

[0027] Output the steady-state risk pressure value set of each node.

[0028] Optionally, the process of generating the label path chain includes:

[0029] Construct a label graph containing predefined sensitive label nodes and their hierarchical relationships;

[0030] Establish soft connection edges between the semantic risk graph and the label graph, and the edge weight is calculated based on the node embedding similarity;

[0031] Starting from the high-risk node, a cross-graph path is searched to generate a path chain containing the starting node, intermediate nodes, and target label nodes.

[0032] In a second aspect, the present invention further provides a multimodal data sensitivity grading system based on semantic risk graph diffusion perception, which is used to implement a multimodal data sensitivity grading method based on semantic risk graph diffusion perception, and the system includes:

[0033] The multimodal preprocessing module is used to perform modality recognition and structural processing on the input multimodal data, screen cross-modal segment combinations based on semantic coupling, and construct semantic risk units;

[0034] The feature fusion module is used to extract the semantic embedding representation of each modal segment in the semantic risk unit, map it to the shared semantic space, and perform weighted aggregation to generate a fused representation vector;

[0035] The semantic graph construction module is used to construct a semantic risk graph based on semantic similarity, modality co-occurrence frequency, entity consistency, and contextual relationships, using semantic risk units as nodes.

[0036] The risk diffusion module is used to initialize the risk pressure value for the nodes of the semantic risk graph, simulate the risk propagation based on the graph structure using the heat diffusion mechanism, and obtain the steady-state risk value of the node;

[0037] The level assessment module is used to fuse the steady-state risk values ​​of all nodes to construct a sample-level risk vector, input the scoring function and output the sensitivity level prediction result;

[0038] The interpretable output module is used to introduce the sensitive label map and jointly model it with the semantic risk map, generate a label path chain through semantic path matching, and output structured interpretation results.

[0039] In a third aspect, the present invention further provides a computer terminal device, comprising:

[0040] one or more processors;

[0041] a memory, coupled to the processor, for storing one or more programs;

[0042] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the multimodal data sensitivity grading method based on semantic risk graph diffusion perception in the above-mentioned first aspect.

[0043] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multimodal data sensitivity grading method based on semantic risk graph diffusion perception in the above-mentioned first aspect are implemented.

[0044] In a fifth aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the multimodal data sensitivity grading method based on semantic risk graph diffusion perception in the above-mentioned first aspect.

[0045] Compared with the prior art, the present invention has the following advantages and technical effects:

[0046] The present invention provides a multimodal data sensitivity grading method and system based on semantic risk graph diffusion perception. The present invention realizes the precise association of cross-modal sensitive information by constructing semantic risk units, uses semantic risk graphs to explicitly model the collaborative relationship between multimodal segments, and combines the heat diffusion mechanism to dynamically simulate the risk propagation process in the context, which significantly improves the accuracy and robustness of sensitivity identification in complex scenarios. By jointly matching the sensitive label map with the semantic risk graph, a structured explanation chain is generated, which realizes the traceability of the judgment results and compliance audit support. The present invention effectively solves the problems of weak cross-modal linkage capabilities, insufficient context modeling and poor interpretability of traditional technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 This is an overall flow chart of a multimodal data sensitivity grading method and system based on semantic risk graph diffusion perception according to an embodiment of the present invention;

[0049] Figure 2 This is a diagram of the multimodal semantic feature extraction and SRU construction framework of an embodiment of the present invention;

[0050] Figure 3 This is a framework diagram of the semantic risk graph construction and diffusion process of an embodiment of the present invention;

[0051] Figure 4 This is a label path interpretability framework diagram of an embodiment of the present invention;

[0052] Figure 5 This is a framework diagram of a multimodal sensitivity detection system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0054] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment provides a multimodal data sensitivity grading method based on semantic risk graph diffusion perception, including:

[0057] Perform modality recognition and structural processing on the input multimodal data, screen cross-modal segment combinations based on semantic coupling, and construct semantic risk units;

[0058] Extract the semantic embedding representation of each modality segment in the semantic risk unit, map it to the shared semantic space and perform weighted aggregation to generate a fusion representation vector;

[0059] Taking semantic risk units as nodes, a semantic risk graph is constructed based on semantic similarity, modality co-occurrence frequency, entity consistency and contextual relationship;

[0060] Initialize the risk pressure value for the nodes of the semantic risk graph, simulate the risk propagation based on the graph structure using the heat diffusion mechanism, and obtain the steady-state risk value of the node;

[0061] The steady-state risk values ​​of all nodes are integrated to construct a sample-level risk vector, which is then input into the scoring function to output the sensitivity level prediction result.

[0062] A sensitive label graph is introduced and jointly modeled with the semantic risk graph. A label path chain is generated through semantic path matching, and structured explanation results are output.

[0063] The specific steps include:

[0064] S1: Performs modality recognition and structural processing on the input multimodal data (including text, images, audio, and video), screens cross-modal segment combinations based on semantic coupling, and constructs a semantic risk unit (SRU) as the minimum sensitive context unit;

[0065] S2: Call the modality-adapted pre-trained model to extract the semantic embedding representation of each modality segment in the SRU, unify it into a shared semantic space through linear mapping, and use a weighted aggregation strategy to generate the fused representation vector of the SRU;

[0066] S3: Using all SRUs as nodes, we construct graph edge weights based on semantic similarity, modal co-occurrence frequency, entity consistency, and contextual relationships to form a multi-relational semantic risk graph (SRG) to explicitly depict the potential semantic risk collaborative structure.

[0067] S4: Initialize the risk pressure value for each SRU node and use the heat diffusion mechanism based on the semantic graph structure to model the risk propagation process in the graph, ultimately obtaining the steady-state risk expression value for each node;

[0068] S5: Perform weighted fusion on the steady-state pressure values ​​of all SRU nodes in the same original sample to construct a sample-level risk vector, which is then input into the trained sensitivity level scoring function to output a discrete or continuous risk level prediction result.

[0069] S6: Introduce a sensitive label graph and jointly model it with a semantic risk graph. Generate a label path chain through cross-graph semantic path matching to achieve a structured interpretation output of the sensitivity level determination results, supporting path-level traceability and causal chain analysis.

[0070] As an implementation method of this embodiment, the process of constructing a semantic risk unit includes:

[0071] Align multimodal data along a unified time axis to generate a triple representation containing modality type, timestamp, and embedded features;

[0072] Calculate the semantic similarity, temporal error and contextual consistency indicators of any two modal segments;

[0073] Based on a preset threshold, segment pairs that meet the semantic coupling condition are screened to form a candidate combination set;

[0074] The risk relevance score of candidate combinations is calculated based on semantic fusion strength, entity consistency and sensitive label directionality, and combinations with scores exceeding the threshold are screened to construct semantic risk units.

[0075] For details, please refer to the attached Figure 2As shown, this embodiment provides an implementation process for multimodal semantic feature extraction and semantic risk unit (SRU) construction. After feature extraction and alignment processing, the data from each modality enters the SRU construction module to form the structured input nodes required for subsequent semantic risk graph modeling. This process includes the following steps:

[0076] S1.1: Perform modality recognition and structural processing on the input raw multimodal data, parsing heterogeneous modal data such as text, images, audio, and video into a standardized set of segments. Each segment generates a semantic embedding vector through an adapted modal feature extraction model, and uniformly aligns them based on the time axis to form a triple:

[0077] ;

[0078] in is the modal type, is the timestamp of the fragment, is the modality embedding feature vector. The generation model and method of the embedding vector will be described in detail in step S2.

[0079] S1.2: For any two modal segments and , respectively based on the embedding vector , timestamp and contextual co-reference information to calculate their semantic similarity , time error , consistency with context , used to evaluate the semantic coupling strength between segments. Semantic similarity measures the similarity between two modal segments in the semantic space and is calculated using the cosine similarity of the embedding vectors. The formula is as follows:

[0080] ;

[0081] in are the semantic embedding vectors of the fragments, , the larger the value, the closer the semantics.

[0082] Time alignment error refers to the degree of offset between two modal segments on the time axis, reflecting whether they occur in similar time periods. Its calculation formula is as follows:

[0083] ;

[0084] in , Separate fragments 、 timestamp, The smaller the value, the closer the time interval between the two segments is.

[0085] Contextual consistency is used to measure whether two modal segments share similar or identical semantic backgrounds, measured by co-referencing entities, consistent events, and shared labels, and is defined as follows:

[0086] ;

[0087] in , Separate fragments 、 The set of named entities identified in , the closer it is to 1, the more consistent the entity is.

[0088] S1.3: Setting similarity threshold , time alignment error threshold and contextual consistency threshold When the fragment combination meets the conditions: When and There is a strong correlation in time, semantics and context, forming a cross-modal combination candidate pair and adding it to the candidate set , as a potential sensitive context structure unit.

[0089] S1.4: To further screen candidate combinations, Calculate its portfolio risk correlation score , which is used to measure the sensitivity risk of cross-modal fragment combinations. The specific scoring function is defined as follows:

[0090] ;

[0091] in express and The fusion tightness in the semantic vector space is achieved through attention scores and consistency measurement after vector aggregation; express and The similarity of the included named entities is calculated based on the normalization of the overlap of shared entity types or labels; It represents the probability that the combination points to the sensitive label category, which is generated by the trained classification model and label mapping function. are the weighting coefficients of the three types of indicators.

[0092] S1.5: Set scoring thresholds , when the candidate combination Risk score satisfy When it is determined to have a high risk tendency, it is constructed as a semantic risk unit SRU, which serves as the basic node input in the subsequent semantic graph structure modeling process.

[0093] As an implementation method of this embodiment, the process of generating the fused representation vector includes:

[0094] Call the pre-trained model adapted to the modality type to extract the semantic feature representation of each modality segment;

[0095] Mapping features to a unified semantic space through a modality-specific projection matrix;

[0096] A weighted fusion strategy is adopted for the multimodal embedding representations within the same semantic risk unit to generate a cross-modal semantic fusion vector.

[0097] Specifically, the above process includes:

[0098] S2.1: For each multimodal segment contained in SRU, the pre-trained model adapted to its modality type is called to extract semantic feature representation. The original input is , and its corresponding feature extractor is , then the modal eigenvector is obtained The specific implementation includes: the text modality uses the BERT model, with the input being the text fragment after word segmentation and the output being the contextual semantic embedding vector, and the [CLS] bit output is extracted as the fragment representation; the image modality first performs target region detection and cropping through YOLOv5 before processing, and then inputs the cropped region into the CLIP model to output the visual semantic embedding; the audio modality extracts the spectrogram after dividing the audio fragment into frames, inputs it into the VGGish model, and outputs the acoustic feature representation; the video modality divides the continuous frames into stable action segments of fixed length, inputs them into the I3D (Inflated 3D ConvNet) model, and extracts the action semantic vector for representing the temporal motion semantics.

[0099] S2.2: To align multimodal semantic features in a shared representation space, design a linear projection matrix for each modality type. , used to map the original features to a unified semantic space. The mapping formula is as follows:

[0100] ;

[0101] in, Indicates modality Middle The embedded representation of each fragment in a unified semantic space, is a modality-specific feature extractor, For modal A specific linear mapping matrix, Embedding modal fragments in a unified semantic space.

[0102] S2.3: For multiple modal segments in the same SRU, the corresponding embedding representation set is denoted as:

[0103] ;

[0104] The weighted fusion strategy is used to integrate the embedding vectors in the set into a unified representation vector of SRU , this process comprehensively considers factors such as modal confidence and context contribution. The fusion formula is as follows:

[0105] ;

[0106] in, Indicates the The weight coefficients of each modal segment in the fusion process are introduced in the training phase. In the training mode, a learnable multi-layer perceptron is introduced to calculate the score of each modal segment, which is obtained through softmax normalization. In the non-training mode, fixed weights are set according to a priori.

[0107] The final fusion As the only node representation of SRU, it has a unified cross-modal semantic representation capability and is input into the subsequent graph construction and risk reasoning modules.

[0108] As an implementation method in this embodiment, the process of constructing a semantic risk map includes:

[0109] Map the semantic risk units to nodes of the graph, and use their fused representation vectors as node attributes;

[0110] Calculate the edge weights between nodes based on semantic similarity, co-referencing entity overlap, temporal relationship, sensitive label consistency, and modal co-occurrence frequency;

[0111] Generate an adjacency matrix of multi-relation expressions to form a semantic risk graph.

[0112] Specifically, the above process includes:

[0113] Refer to the attached Figure 3 As described above, this embodiment provides a mechanism for constructing a semantic risk graph and processing risk pressure diffusion and propagation. Using semantic risk units (SRUs) as nodes, a multi-relationship fusion graph structure is constructed. A graph neural network modeling method based on heat diffusion is introduced to simulate the propagation and evolution of potential risks in a semantic context, ultimately completing node-level risk characterization and sample-level sensitivity classification. The specific steps include:

[0114] S3.1: Each semantic risk unit Mapped to a node in the graph , whose characteristics are represented as node attribute vectors The set of vertices of all SRU graphs , the corresponding node attribute matrix is .

[0115] S3.2: Constructing the edge set in the graph , analyze any two nodes and The multi-dimensional coupling relationship between It is constructed based on the following five categories of features:

[0116] Semantic similarity : It is defined as the cosine similarity between node feature vectors, and the calculation formula is as follows:

[0117] ;

[0118] in is the fusion representation vector of the node, represents the L2 norm.

[0119] Co-referential entity overlap :measure and The degree of overlap of the set of included named entities is defined as follows:

[0120] ;

[0121] The higher the value, the more consistent the semantic orientation of the two SRUs.

[0122] Time sequence indicator :Used to indicate and The temporal relationship that occurs in the original sample is defined as:

[0123] ;

[0124] in Respectively and The timestamp of the earliest fragment in the time series is converted into the closeness in the interval [0,1]. The closer the two SRU times are, the closer they are. The closer it is to 1.

[0125] Sensitive label consistency : Indicates whether two nodes point to the same sensitive label. It is defined as follows:

[0126] ;

[0127] in They are and The predicted label of Represents a collection of sensitive categories.

[0128] Modal co-occurrence frequency :measure and The degree of overlap of the modal types is defined as:

[0129] ;

[0130] in and These are all sets of modal types included in the SRU. A higher value indicates more consistent modal coverage.

[0131] The above features include five types of heterogeneous relationship features, including semantic similarity, co-referencing entity overlap, temporal relationship, sensitive label consistency and modal co-occurrence frequency, and the edge weights are calculated by weighted fusion. , the fusion formula is as follows:

[0132] ;

[0133] in, is the edge weight fusion coefficient, satisfying ,During the training phase, the fusion coefficient is set as a learnable parameter, which participates in the back-propagation update of the final prediction loss function after semantic graph risk propagation.

[0134] S3.3: Based on the above edge weight definition, generate the adjacency matrix of the graph , is the number of nodes in the graph, where Representation node and The connection strength between them. The final constructed semantic risk graph can be represented as a triple , which has the characteristics of multimodal fusion, context alignment and semantic coupling, and provides a structural basis for subsequent graph diffusion propagation and risk grading.

[0135] As an implementation method in this embodiment, the process of the heat diffusion mechanism includes:

[0136] Initialize the node risk pressure value according to label confidence, modal coverage completeness, semantic ambiguity and entity recognition confidence;

[0137] Iteratively update the pressure value based on the relationship between adjacent nodes until the average change is lower than the convergence threshold;

[0138] Output the steady-state risk pressure value set of each node.

[0139] Specifically, the above process includes:

[0140] S4.1: Create a semantic risk map Each node Initialize risk pressure value , as the starting state of graph diffusion propagation. The initial pressure value is obtained by weighting multi-dimensional attributes, mainly including label confidence : Indicates a node The confidence probability of being predicted as a sensitive label comes from the softmax output of the sensitivity discrimination model; modal coverage completeness :express The richness of the covered modalities is defined as the ratio of the number of actually covered modalities to the total number of available modalities, with a value range of [0,1]; the context semantic fuzziness : Indicates the node representation vector The semantic uncertainty of is measured by vector entropy, variance and sentence vector density estimation indicators. Higher values ​​indicate less certainty about the information; and the confidence level of named entity recognition : represents the average confidence of the named entity recognition model output in SRU, which is used to measure the quality of entity information. The initial pressure calculation formula is as follows:

[0141] ;

[0142] in , which is the weighting coefficient of each attribute factor, is set based on the importance of different modes or the distribution characteristics of the training set.

[0143] S4.2: After the initial pressure is assigned, multiple rounds of iterative propagation are performed based on the graph's adjacency structure to simulate the transmission of semantic risk in the contextual network. The diffusion mechanism is analogous to the heat conduction model, controlling the proportion of pressure from each node that spreads to adjacent nodes. The diffusion iteration formula is as follows:

[0144] ;

[0145] in Representation node The set of adjacent nodes of Represents a slave node To Node The propagation weight coefficient can be calculated based on the edge weight Normalized to get:

[0146] ;

[0147] This strategy ensures that the influence of neighboring nodes on the current node is distributed according to their relative strength in each diffusion.

[0148] S4.3: To ensure that the graph diffusion process converges within a limited number of rounds, a stability criterion is set. If the average pressure change of all nodes between two consecutive rounds of propagation is lower than the convergence threshold , then terminate the iteration, the judgment formula is as follows:

[0149] ;

[0150] After the convergence conditions are met, the steady-state risk pressure value set of each node is output:

[0151] ;

[0152] in For nodes The final risk expression is used to support subsequent sample-level sensitivity risk aggregation and level determination.

[0153] S5.1: For each original sample, collect the steady-state risk pressure values ​​of all SRU nodes it contains to form the overall risk expression vector of the sample, which is defined as follows:

[0154] ;

[0155] in Indicates that the sample For the sample The steady-state pressure value of each SRU node, is the number of SRUs in the sample. If the number of SRUs in different samples is inconsistent, vectors can be unified through methods such as average padding, maximum pooling, and fixed-dimensional truncation.

[0156] S5.2: Use the scoring function fitted in the training phase to calculate the sample risk vector Perform linear weighting and normalization to calculate the final sensitivity level score. The scoring function is defined as follows:

[0157] ;

[0158] in, is a fixed weight vector that reflects the relative importance of each SRU pressure to the final level. is the bias term, both of which are determined through supervised learning training; It is the risk score normalization function. The Sigmoid function is used in the binary classification scenario, and the Softmax function is used in the multi-level judgment scenario. The output is the normalized probability distribution of the corresponding level.

[0159] S5.3: Continuous scoring results based on the output of the trained model , mapping it to a discrete sensitivity level label , the mapping is based on the set level interval segmentation:

[0160] ;

[0161] in Indicates different sensitivity level labels, such as "non-sensitive", "low-sensitive", "medium-sensitive", "high-sensitive", etc.; segmentation threshold The sensitivity level label is obtained through parameter adjustment of the validation set or empirical setting. The final output sensitivity level label not only serves as the judgment result of the system, but also serves as the core basis for multi-module collaboration. Specifically, it is used for risk compliance interface display, supporting result visualization and policy rule docking; triggering automated review strategies to achieve dynamic interception and labeling of sensitive content; serving as input to the semantic path interpretation module, supporting causal traceability and structured interpretation of sensitivity level decisions; and providing risk warnings and preliminary classification suggestions for the manual review process, improving review efficiency and accuracy.

[0162] As an implementation method of this embodiment, the process of generating a label path chain includes:

[0163] Construct a label graph containing predefined sensitive label nodes and their hierarchical relationships;

[0164] Establish soft connection edges between the semantic risk graph and the label graph, and the edge weight is calculated based on the node embedding similarity;

[0165] Starting from the high-risk node, a cross-graph path is searched to generate a path chain containing the starting node, intermediate nodes, and target label nodes.

[0166] Specifically, the above process includes:

[0167] Refer to the attached Figure 4 As mentioned above, this embodiment provides an explainable sensitivity determination mechanism based on semantic path matching, aiming to provide a structured label path chain for the sensitivity level output of each sample and support path-level causal analysis and traceability. The specific process is as follows:

[0168] S6.1: Build a sensitive label map:

[0169] ;

[0170] in Represents the set of label nodes in the sensitive label graph, each node corresponds to a sensitive category, such as "document information"; edge set Represents the hierarchical relationship (superordinate and subordinate), attribute inclusion relationship, or semantic similarity connection between tags; node attribute matrix The embedded features representing the label nodes are used to generate label semantic vectors through a pre-trained language model. This label map can be constructed based on national standards, industry specifications, or sensitive word ontology, with a clear semantic classification and inheritance structure, providing a foundation for subsequent label matching.

[0171] S6.2: To achieve semantic mapping between SRU nodes and label nodes, a cross-graph “soft connection edge” is introduced between the semantic risk graph and the sensitive label graph, and a joint graph structure is constructed:

[0172] ;

[0173] in is the SRU node in the semantic risk graph, Represents the set of label nodes in the sensitive label graph; is the edge set of the joint graph; is the fusion feature of SRU node; is a set of soft connection edges; the edge weight of each soft connection edge is defined as:

[0174] ;

[0175] in is the semantic vector of the SRU node, Embedding representation for label nodes, is the cosine similarity function in the embedding space. Set the similarity threshold , remove invalid connections with too low edge weights to reduce redundant paths in the graph.

[0176] S6.3: Based on joint graph Starting from each high-risk node, we perform a cross-graph path search to discover the shortest or most weighted path to the label node. The resulting path chain (Label Path Chain) consists of the following components: a starting node (high-risk SRU), a sequence of intermediate connecting nodes (semantic / modal transitions), a sequence of connecting edge weights, and a target label node (e.g., "document information"). Each path can be formalized as:

[0177] ;

[0178] in The starting node of the path is the identified high-risk SRU node; Indicates the intermediate nodes in the path, which are other SRU nodes or intermediate label nodes; Represents the endpoint node of the path, corresponding to the hit sensitive label (such as "document information category"); this structured path chain is used to describe the semantic link of risk triggering, which helps to model the interpretability of the judgment process.

[0179] S6.4: For each sample's output sensitivity level prediction result, the corresponding label path chain is attached as a structured interpretation result, supporting path-level visual traceability and sensitivity trigger factor analysis. The output format is as follows:

[0180] Image node: [ID photo];

[0181] Text node: ["The certificate number is 330XXXX"];

[0182] Tag trigger: [certificate information category];

[0183] Co-occurrence with video node: [patient self-report video];

[0184] Judgment level: [Highly sensitive];

[0185] The path displays the causal chain of the judgment results, key modal trigger points, label docking logic and contextual collaborative evidence, supporting path-level traceability analysis, semantic trigger factor interpretation, manual audit verification assistance and transparent display of results compliance.

[0186] S6.5: In order to further improve the path quality and interpretability ranking, the graph attention network (GAT) is introduced to the joint graph Perform embedding learning to learn the connection strength and label association between nodes across the graph. The embedding is represented as follows:

[0187] ;

[0188] Graph neural networks can adaptively learn the importance weights of connecting edges, optimize the quality of path generation, and use them for path priority sorting, thereby improving the availability and credibility of the final output.

[0189] Based on this, an embodiment of the present invention provides a multimodal data sensitivity grading method based on semantic risk graph diffusion perception, which systematically solves the key problems of existing multimodal sensitivity identification methods such as insufficient semantic fusion depth, weak context linkage modeling capabilities, and lack of interpretability of sensitivity judgment results. By constructing cross-modal semantic risk units, fusing graph structure expressions, introducing risk pressure diffusion mechanisms and sensitive label graph alignment strategies, the present invention realizes a structured and traceable multimodal sensitivity grading judgment process. Based on the advantages of the above methods, the present invention designs a multimodal data sensitivity grading system consisting of six core modules. The multimodal data preprocessing module is used to receive text, image, audio and video data, complete modal identification, time alignment and segment standardization, and generate candidate semantic units; the feature extraction and unified representation module uses a pre-trained model to extract the semantic embedding of each modal segment and map it to a shared semantic space; the semantic risk graph construction module constructs a heterogeneous graph structure based on semantic similarity, entity co-reference and other relationships between segments; the graph diffusion reasoning module simulates the risk propagation process through a heat diffusion mechanism to generate node-level risk expressions; the sensitivity level assessment module aggregates node risk vectors and outputs sample-level sensitivity levels through a scoring function; the interpretable output module constructs a sensitive label map and generates a label path chain, providing structured judgment interpretation and causal link traceability capabilities.

[0190] Example 2

[0191] In this embodiment, a computer terminal device is provided, including:

[0192] one or more processors;

[0193] a memory, coupled to the processor, for storing one or more programs;

[0194] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above-mentioned multimodal data sensitivity grading method based on semantic risk graph diffusion perception.

[0195] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multimodal data sensitivity grading method based on semantic risk graph diffusion perception are implemented.

[0196] In this embodiment, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the above-mentioned multimodal data sensitivity grading method based on semantic risk graph diffusion perception.

[0197] In this embodiment, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned multimodal data sensitivity grading method based on semantic risk graph diffusion perception.

[0198] The above program can be executed in a processor or stored in a memory (or computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0199] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a block or multiple blocks can be implemented by different modules corresponding to different steps.

[0200] This embodiment provides such a device or system. The system is called a multimodal data sensitivity grading system based on semantic risk graph diffusion perception, and includes:

[0201] The multimodal preprocessing module is used to perform modality recognition and structural processing on the input multimodal data, screen cross-modal segment combinations based on semantic coupling, and construct semantic risk units;

[0202] The feature fusion module is used to extract the semantic embedding representation of each modal segment in the semantic risk unit, map it to the shared semantic space, and perform weighted aggregation to generate a fused representation vector;

[0203] The semantic graph construction module is used to construct a semantic risk graph based on semantic similarity, modality co-occurrence frequency, entity consistency, and contextual relationships, using semantic risk units as nodes.

[0204] The risk diffusion module is used to initialize the risk pressure value for the nodes of the semantic risk graph, simulate the risk propagation based on the graph structure using the heat diffusion mechanism, and obtain the steady-state risk value of the node;

[0205] The level assessment module is used to fuse the steady-state risk values ​​of all nodes to construct a sample-level risk vector, input the scoring function and output the sensitivity level prediction result;

[0206] The interpretable output module is used to introduce the sensitive label map and jointly model it with the semantic risk map, generate a label path chain through semantic path matching, and output structured interpretation results.

[0207] As an implementation in this embodiment, the multimodal preprocessing module includes:

[0208] The fragment alignment unit is used to align multimodal data along a unified time axis and generate a triple representation containing modality type, timestamp, and embedded features;

[0209] A coupling screening unit is used to calculate the semantic similarity, temporal error, and contextual consistency indicators of any two modal segments and screen candidate combinations based on preset thresholds;

[0210] The risk unit generation unit is used to calculate the risk relevance score based on the semantic fusion strength, entity consistency and sensitive label directionality, and construct a semantic risk unit.

[0211] As an implementation method of this embodiment, the feature fusion module includes:

[0212] A modal feature extraction unit is used to call a pre-trained model adapted to the modality type to extract the semantic feature representation of each modal segment;

[0213] A spatial mapping unit, used to map features into a unified semantic space through a modality-specific projection matrix;

[0214] The weighted aggregation unit is used to adopt a weighted fusion strategy for the multimodal embedding representation within the same semantic risk unit to generate a cross-modal semantic fusion vector.

[0215] As an implementation method in this embodiment, the semantic graph construction module includes:

[0216] A node mapping unit is used to map semantic risk units to nodes of a graph, and its fused representation vector is used as a node attribute;

[0217] Edge weight calculation unit, used to calculate the edge weights between nodes based on semantic similarity, co-referencing entity overlap, temporal relationship, sensitive label consistency, and modal co-occurrence frequency;

[0218] The graph generation unit is used to generate an adjacency matrix of multi-relational expressions to form a semantic risk graph.

[0219] As an implementation in this embodiment, the risk diffusion module includes:

[0220] Pressure initialization unit, used to initialize the node risk pressure value according to label confidence, modal coverage completeness, semantic fuzziness and entity recognition confidence;

[0221] An iterative propagation unit, used to iteratively update the pressure value based on the relationship between adjacent nodes until the average change is lower than the convergence threshold;

[0222] The steady-state output unit is used to output the steady-state risk pressure value set of each node.

[0223] As an implementation method of this embodiment, the interpretable output module includes:

[0224] A label graph construction unit, used to construct a label graph containing predefined sensitive label nodes and their hierarchical relationships;

[0225] Cross-graph connection unit, used to establish soft connection edges between the semantic risk graph and the label graph. The edge weight is calculated based on the node embedding similarity.

[0226] The path generation unit is used to search for cross-graph paths starting from high-risk nodes and generate a path chain containing a start node, intermediate nodes, and a target label node.

[0227] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0228] Through the above implementation, the problem of multimodal data sensitivity classification based on semantic risk graph diffusion perception in the related art is solved, thereby ensuring that the problems existing in the existing technology are solved.

[0229] Example 3

[0230] Refer to the attached Figure 5 This embodiment provides a multimodal data sensitivity grading method and system based on semantic risk graph diffusion perception, including the following modules:

[0231] The multimodal data preprocessing module is used to receive text, image, audio and video data, perform modality recognition, time alignment and segment normalization, and generate candidate semantic units;

[0232] The feature extraction and unified representation module uses a pre-trained model to extract the semantic embedding of each modal segment and maps it to a shared semantic space;

[0233] The semantic risk graph construction module: constructs a heterogeneous graph structure based on semantic similarity, entity co-reference and other relationships between segments;

[0234] The graph diffusion reasoning module simulates the risk propagation process through the heat diffusion mechanism and generates node-level risk expressions;

[0235] The sensitivity level assessment module aggregates node risk vectors and outputs sample-level sensitivity levels through a scoring function.

[0236] The explainable output module constructs a sensitive label map and generates a label path chain, providing structured judgment interpretation and causal link traceability capabilities.

[0237] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A multimodal data sensitivity grading method based on semantic risk graph diffusion perception, characterized by: The following steps are involved: Perform modality recognition and structural processing on the input multimodal data, screen cross-modal segment combinations based on semantic coupling, and construct semantic risk units; Extract the semantic embedding representation of each modality segment in the semantic risk unit, map it to the shared semantic space and perform weighted aggregation to generate a fusion representation vector; Taking semantic risk units as nodes, a semantic risk graph is constructed based on semantic similarity, modality co-occurrence frequency, entity consistency and contextual relationship; Initialize the risk pressure value for the nodes of the semantic risk graph, simulate the risk propagation based on the graph structure using the heat diffusion mechanism, and obtain the steady-state risk value of the node; The steady-state risk values ​​of all nodes are integrated to construct a sample-level risk vector, which is then input into the scoring function to output the sensitivity level prediction result. A sensitive label graph is introduced and jointly modeled with the semantic risk graph. A label path chain is generated through semantic path matching, and structured explanation results are output.

2. The method according to claim 1, characterized in that The process of constructing the semantic risk unit includes: Align multimodal data along a unified time axis to generate a triple representation containing modality type, timestamp, and embedded features; Calculate the semantic similarity, temporal error and contextual consistency indicators of any two modal segments; Based on a preset threshold, segment pairs that meet the semantic coupling condition are screened to form a candidate combination set; The risk relevance score of candidate combinations is calculated based on semantic fusion strength, entity consistency and sensitive label directionality, and combinations with scores exceeding the threshold are screened to construct semantic risk units.

3. The method according to claim 1, characterized in that The process of generating the fused representation vector includes: Call the pre-trained model adapted to the modality type to extract the semantic feature representation of each modality segment; Mapping features to a unified semantic space through a modality-specific projection matrix; A weighted fusion strategy is adopted for the multimodal embedding representations within the same semantic risk unit to generate a cross-modal semantic fusion vector.

4. The method according to claim 1, wherein The process of constructing the semantic risk map includes: Map the semantic risk units to nodes of the graph, and use their fused representation vectors as node attributes; Calculate the edge weights between nodes based on semantic similarity, co-referencing entity overlap, temporal relationship, sensitive label consistency, and modal co-occurrence frequency; Generate an adjacency matrix of multiple relation expressions to form a semantic risk graph.

5. The method according to claim 1, wherein The process of the heat diffusion mechanism includes: Initialize the node risk pressure value according to label confidence, modal coverage completeness, semantic ambiguity and entity recognition confidence; Iteratively update the pressure value based on the relationship between adjacent nodes until the average change is lower than the convergence threshold; Output the steady-state risk pressure value set of each node.

6. The method according to claim 1, characterized in that The process of generating a label path chain includes: Construct a label graph containing predefined sensitive label nodes and their hierarchical relationships; Establish soft connection edges between the semantic risk graph and the label graph, and the edge weight is calculated based on the node embedding similarity; Starting from the high-risk node, a cross-graph path is searched to generate a path chain containing the starting node, intermediate nodes, and target label nodes.

7. A multimodal data sensitivity grading system based on semantic risk graph diffusion perception, characterized by: The system comprises: The multimodal preprocessing module is used to perform modality recognition and structural processing on the input multimodal data, screen cross-modal segment combinations based on semantic coupling, and construct semantic risk units; The feature fusion module is used to extract the semantic embedding representation of each modal segment in the semantic risk unit, map it to the shared semantic space, and perform weighted aggregation to generate a fused representation vector; The semantic graph construction module is used to construct a semantic risk graph based on semantic similarity, modality co-occurrence frequency, entity consistency, and contextual relationships, using semantic risk units as nodes. The risk diffusion module is used to initialize the risk pressure value for the nodes of the semantic risk graph, simulate the risk propagation based on the graph structure using the heat diffusion mechanism, and obtain the steady-state risk value of the node; The level assessment module is used to fuse the steady-state risk values ​​of all nodes to construct a sample-level risk vector, input the scoring function and output the sensitivity level prediction result; The interpretable output module is used to introduce the sensitive label map and jointly model it with the semantic risk map, generate a label path chain through semantic path matching, and output structured interpretation results.

8. A computer terminal device, characterized in that: include: one or more processors; a memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images