Method and system for secure identification and intelligent classification processing of multi-modal data

Abstract

The present application relates to the technical field of data processing, and more particularly to a kind of multi-modal data security identification and intelligent classification processing method and system, comprising obtaining heterogeneous source multi-modal data and extracting each modal feature vector to carry out security attribute labeling, construct cross-modal association graph and calculate sensitive source influence index, identify risk convergence node and determine risk convergence coefficient, construct risk propagation matrix to identify leading risk propagation direction, after adjusting the edge weight of cross-modal association graph, carry out feature fusion and classification processing, improve the accuracy and efficiency of multi-modal data security identification and classification.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for secure identification and intelligent classification of multimodal data. Background Technology

[0002] In the current field of multimodal data processing, conventional practices typically involve extracting features from heterogeneous data from different sources, and then simply concatenating or weighting the extracted feature vectors before inputting them into a classifier for recognition. To ensure data security, some solutions add labels or classify privacy levels after feature extraction, and then perform filtering or anonymization based on these labels. However, these methods often treat the features of each modality as independent entities, performing only linear combinations during the fusion stage, without deeply exploring the semantic relationships and interaction patterns between modalities.

[0003] The primary drawback of conventional approaches is that they ignore the inherent semantic differences and cross-modal correspondences in multimodal data. Because different modalities (such as text, images, and audio) have different expressive spaces and information densities, direct splicing or simple weighting easily introduces noise, destroying the inherent structured information of each modality and thus affecting the accuracy of subsequent classification tasks. Especially when data contains sensitive attributes (such as personal identity or biometrics), relying solely on single-modal labels cannot reflect the coupling of risks across modalities, leading to blind spots in security identification.

[0004] Another shortcoming is the lack of analysis on the propagation paths and risk convergence effects of sensitive data. Existing technologies typically only assess security for single data samples or single-modal features, without constructing modeling methods to quantify the transmission and accumulation of sensitive information across modalities. For example, when multiple sensitive data nodes form convergence nodes through implicit relationships, their potential risks may be far greater than the sum of the risks of individual nodes. However, conventional methods cannot identify such risk convergence and the dominant propagation direction, making the formulation of security protection strategies lack specificity and easily overlooking the need for differentiated handling of cross-modal risks.

[0005] In summary, existing multimodal security identification and classification methods have significant shortcomings in modality association mining and risk propagation modeling, making it difficult to meet the dual requirements of accurate protection and intelligent classification of sensitive information in complex and heterogeneous data environments. Summary of the Invention

[0006] The embodiments of the present invention provide a method and system for secure identification and intelligent classification of multimodal data, which can solve the problems in the prior art.

[0007] A first aspect of this invention provides a method for secure identification and intelligent classification of multimodal data, comprising:

[0008] Acquire heterogeneous source multimodal data and extract feature vectors for each modality. Label each feature vector with security attributes to generate a set of feature vectors with security labels.

[0009] A cross-modal association graph is constructed based on the semantic similarity between feature vectors of different modalities in a feature vector set with security labels;

[0010] In the cross-modal association graph, the feature vector with security label is used as the sensitive source node. The set of neighborhood nodes and modal distribution characteristics of each sensitive source node are calculated. The influence index of the sensitive source is determined based on the set of neighborhood nodes and modal distribution characteristics.

[0011] Identify nodes that are associated with multiple sensitive source nodes in the cross-modal association graph as risk aggregation nodes, count the security type of the sensitive source nodes associated with each risk aggregation node, and determine the risk aggregation coefficient based on the security type;

[0012] A risk propagation matrix is ​​constructed based on the influence index of sensitive sources and the risk convergence coefficient. The risk propagation matrix is ​​then decomposed by eigenvalue to identify the dominant risk propagation direction and generate a risk propagation vector.

[0013] The directional decay factor of each edge in the cross-modal association graph is calculated based on the risk propagation vector. The edge weights are adjusted based on the directional decay factor to generate an adjusted cross-modal association graph. Feature fusion and classification are then performed based on the adjusted cross-modal association graph.

[0014] Acquire heterogeneous multimodal data and extract feature vectors for each modality. Label each feature vector with security attributes to generate a set of feature vectors with security labels, including:

[0015] Semantic space clustering analysis is performed on heterogeneous source multimodal data to identify the cluster center regions of heterogeneous source multimodal data in the semantic space as sensitive semantic regions;

[0016] Extract the feature vectors of each modality from heterogeneous source multimodal data, calculate the semantic distance between each feature vector and the center of the sensitive semantic region, and determine the semantic sensitivity value of each feature vector based on the semantic distance;

[0017] Each modal feature vector is mapped to a heterogeneous modal space through modal transformation to obtain a cross-modal mapped feature vector. The cross-modal mapped feature vector is then mapped back to the original modal space through inverse modal transformation to obtain a reconstructed feature vector. The feature difference degree between each modal feature vector and the reconstructed feature vector is calculated. Based on the feature difference degree, the cross-modal mapping stability value of each modal feature vector is determined.

[0018] Security attributes are labeled for each modality feature vector. The semantic sensitivity value and cross-modal mapping stability value are fused to calculate the security risk value of each modality feature vector. Security label levels are assigned to each modality feature vector according to the security risk value, and a set of feature vectors with security labels is generated.

[0019] Constructing a cross-modal association graph based on the semantic similarity between feature vectors of different modalities in a feature vector set with security labels includes:

[0020] Extract the security label level information of each modal feature vector from the feature vector set with security labels, and perform security sensitivity weighting on each modal feature vector according to the security label level information, and assign a security sensitivity weight coefficient to each modal feature vector;

[0021] Semantic encoding projection is performed on each modality feature vector, and each modality feature vector is projected onto a unified semantic space to obtain a unified semantic representation vector. The vector cosine similarity between the unified semantic representation vectors of different modality feature vectors in the unified semantic space is calculated to obtain the initial semantic similarity between different modality feature vectors.

[0022] Security modulation of the initial semantic similarity is performed based on the security sensitivity weight coefficient. The security sensitivity weight coefficients corresponding to different modal feature vectors are multiplied with the initial semantic similarity to obtain the security-modulated semantic similarity.

[0023] Each modal feature vector in the set of feature vectors with security labels is used as a node in the cross-modal association graph. Edge connections are established between different modal feature vector pairs whose semantic similarity after security modulation exceeds a preset similarity threshold. The semantic similarity after security modulation is used as the weight value of the edge connection to generate the cross-modal association graph.

[0024] In the cross-modal association graph, the feature vectors with security labels are used as sensitive source nodes. The neighborhood node set and modal distribution characteristics of each sensitive source node are calculated. Based on the neighborhood node set and modal distribution characteristics, the sensitive source influence indicators are determined, including:

[0025] In the cross-modal association graph, nodes with security-labeled feature vectors whose security label level exceeds the preset security level threshold are extracted as sensitive source nodes. For each sensitive source node, multi-hop graph expansion is performed along the edge connection in descending order of weight value to obtain the set of nodes that can be reached by each sensitive source node within a preset expansion step as the neighborhood node set.

[0026] Calculate the shortest path length between each node in the neighborhood node set and the sensitive source node, and use the reciprocal of the shortest path length as the distance decay factor. Assign the distance decay factor as the node weight to each node in the neighborhood node set.

[0027] The sum of node weights for each modality type in the neighborhood node set is counted, and the proportion of the sum of node weights for each modality type to the total weight of the neighborhood node set is calculated as the modality distribution feature. The information entropy of the modality distribution feature is then calculated to obtain the modality distribution uniformity value.

[0028] The total weight of the neighborhood node set is used as the weighted diffusion range value. The weighted diffusion range value is multiplied by the modal distribution uniformity value to obtain the sensitive source influence index.

[0029] Nodes associated with multiple sensitive source nodes in the cross-modal correlation graph are identified as risk convergence nodes. The security type of the sensitive source nodes associated with each risk convergence node is calculated, and the risk convergence coefficient is determined based on the security type, including:

[0030] Traverse all nodes in the cross-modal association graph, count the number of sensitive source nodes directly associated with each node, and identify nodes whose number of directly associated sensitive source nodes exceeds a preset aggregation threshold as risk aggregation nodes;

[0031] Extract the security identification level information of each sensitive source node directly associated with the risk aggregation node, classify each sensitive source node into security types based on the security identification level information, and obtain the security type set of the risk aggregation node;

[0032] Traverse the security types in the security type set, determine the conflict between any two security types according to the preset security type conflict rules, and construct a conflict topology graph by using the security type pairs with conflicting relationships as edges. The nodes of the conflict topology graph are the security types in the security type set.

[0033] Calculate the degree of each node in the conflict topology graph, where the degree is the number of edges connected to the current node. Mark the node with the largest degree as the core conflict node and obtain the security type corresponding to the core conflict node as the core conflict type.

[0034] The number of sensitive source nodes whose security type belongs to the core conflict type among the directly associated sensitive source nodes of the statistical risk aggregation node is used as the risk aggregation coefficient.

[0035] A risk propagation matrix is ​​constructed based on the influence index of sensitive sources and the risk convergence coefficient. The risk propagation matrix is ​​then subjected to eigenvalue decomposition to identify the dominant risk propagation direction, generating a risk propagation vector including:

[0036] Traverse the propagation path from the sensitive source node to the risk convergence node in the cross-modal association graph, extract the modal types of adjacent nodes on the propagation path, determine whether there is a propagation blocking relationship between the modal types of adjacent nodes according to the preset modal compatibility rules, and count the number of adjacent node pairs with a propagation blocking relationship in the propagation path as the number of propagation blocking points.

[0037] A propagation path with zero propagation blockade points is marked as a valid propagation path;

[0038] Extract the sensitive source influence index of sensitive source nodes and the risk convergence coefficient of risk convergence nodes corresponding to effective propagation paths. For each node pair from a sensitive source node to each risk convergence node, count the number of effective propagation paths connecting the node pairs as the path connectivity.

[0039] A risk propagation matrix is ​​constructed based on the sensitive source influence index corresponding to the sensitive source node, the risk convergence coefficient corresponding to the risk convergence node, and the path connectivity of the node pair. The row index of the risk propagation matrix is ​​the sensitive source node, and the column index is the risk convergence node. For each node pair, the corresponding sensitive source influence index, risk convergence coefficient, and path connectivity are used as input parameters to calculate the matrix element value.

[0040] The risk propagation matrix is ​​decomposed into eigenvalues, and the eigenvector corresponding to the largest eigenvalue is used as the risk propagation vector.

[0041] The directional decay factor of each edge in the cross-modal association graph is calculated based on the risk propagation vector. The edge weights are adjusted based on the directional decay factor to generate an adjusted cross-modal association graph. Feature fusion and classification processing are then performed based on the adjusted cross-modal association graph, including:

[0042] Extract the vector component values ​​corresponding to each node in the risk propagation vector. For each edge in the cross-modal association graph, calculate the difference between the vector component values ​​of the starting node and the ending node of the edge. Calculate the directional decay factor of the edge based on the difference.

[0043] The original weight values ​​of each edge in the cross-modal association graph are attenuated based on the directional attenuation factor to obtain the adjusted edge weight values, and the adjusted cross-modal association graph is generated.

[0044] Based on the adjusted cross-modal association graph, the feature vectors of the neighboring nodes of each node are weighted and aggregated according to the adjusted edge weight values ​​to obtain the fused feature vector.

[0045] Construct a category prototype library, calculate the similarity between the fused feature vector and the feature vectors of each category prototype in the category prototype library, and take the category with the highest similarity as the candidate classification result;

[0046] Extract the candidate classification results of the node in the adjusted cross-modal association graph, count the class frequency of the candidate classification results of the neighboring nodes, take the class with the highest class frequency as the dominant class of the neighborhood, and determine whether the candidate classification result of the node is consistent with the dominant class of the neighborhood. If they are consistent, the candidate classification result is taken as the final classification result. If they are inconsistent, the candidate classification result is corrected based on the adjusted edge weight value to obtain the final classification result.

[0047] A second aspect of this invention provides a secure identification and intelligent classification processing system for multimodal data, comprising:

[0048] The multimodal labeling unit is used to acquire heterogeneous source multimodal data and extract feature vectors of each modality, label each feature vector with security attributes, and generate a set of feature vectors with security labels.

[0049] Cross-modal building blocks are used to construct cross-modal association graphs based on the semantic similarity between feature vectors of different modalities in a set of feature vectors with security labels;

[0050] Sensitive source refers to the unit, which is used in the cross-modal association graph to use feature vectors with security labels as sensitive source nodes, calculate the neighborhood node set and modal distribution characteristics of each sensitive source node, and determine the sensitive source influence index based on the neighborhood node set and modal distribution characteristics;

[0051] The risk aggregation unit is used to identify nodes that are associated with multiple sensitive source nodes in the cross-modal association graph as risk aggregation nodes, count the security type of the sensitive source nodes associated with each risk aggregation node, and determine the risk aggregation coefficient based on the security type.

[0052] The risk propagation unit is used to construct a risk propagation matrix based on the influence index of sensitive sources and the risk convergence coefficient, perform eigenvalue decomposition on the risk propagation matrix to identify the dominant risk propagation direction, and generate a risk propagation vector.

[0053] The fusion classification unit is used to calculate the directional decay factor of each edge in the cross-modal association graph based on the risk propagation vector, adjust the edge weights based on the directional decay factor, generate an adjusted cross-modal association graph, perform feature fusion based on the adjusted cross-modal association graph, and perform classification processing.

[0054] A third aspect of the present invention provides an electronic device, comprising:

[0055] processor;

[0056] Memory used to store processor-executable instructions;

[0057] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0058] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0059] This invention employs a cross-modal association graph to structurally model heterogeneous multimodal data. Based on semantic similarity, it associates features of different modalities, effectively capturing implicit security associations between modalities and overcoming the limitations of single-modal recognition. Combined with a risk convergence coefficient, it accurately assesses the cumulative risk of nodes converging multiple security types. These two factors collaboratively construct a risk propagation matrix, depicting the risk diffusion path from a global perspective and providing a more reliable basis for subsequent classification processing. It suppresses ineffective risk diffusion along non-dominant directions and enhances feature discriminability along dominant risk propagation directions, making the adjusted association graph more focused on high-risk association regions. Feature fusion based on the optimized graph structure effectively reduces the interference of noisy modalities on classification decisions while retaining key security semantic information, significantly improving the classification accuracy and generalization ability of multimodal data in security-sensitive scenarios. It adaptively constructs the security association topology between features without requiring manual pre-setting of security rules, reducing reliance on prior knowledge and minimizing false positives and false negatives. This makes it particularly suitable for applications with stringent requirements for data security and classification reliability, such as financial transactions, medical diagnosis, and intelligent security. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the secure identification and intelligent classification processing method for multimodal data according to an embodiment of the present invention.

[0061] Figure 2 This is a flowchart of the cross-modal association graph node classification processing according to an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0064] Figure 1 This is a flowchart illustrating the secure identification and intelligent classification processing method for multimodal data according to an embodiment of the present invention.

[0065] Methods for secure identification and intelligent classification of multimodal data include:

[0066] Acquire heterogeneous source multimodal data and extract feature vectors for each modality. Label each feature vector with security attributes to generate a set of feature vectors with security labels.

[0067] A cross-modal association graph is constructed based on the semantic similarity between feature vectors of different modalities in a feature vector set with security labels;

[0068] In the cross-modal association graph, the feature vector with security label is used as the sensitive source node. The set of neighborhood nodes and modal distribution characteristics of each sensitive source node are calculated. The influence index of the sensitive source is determined based on the set of neighborhood nodes and modal distribution characteristics.

[0069] Identify nodes that are associated with multiple sensitive source nodes in the cross-modal association graph as risk aggregation nodes, count the security type of the sensitive source nodes associated with each risk aggregation node, and determine the risk aggregation coefficient based on the security type;

[0070] A risk propagation matrix is ​​constructed based on the influence index of sensitive sources and the risk convergence coefficient. The risk propagation matrix is ​​then decomposed by eigenvalue to identify the dominant risk propagation direction and generate a risk propagation vector.

[0071] The directional decay factor of each edge in the cross-modal association graph is calculated based on the risk propagation vector. The edge weights are adjusted based on the directional decay factor to generate an adjusted cross-modal association graph. Feature fusion and classification are then performed based on the adjusted cross-modal association graph.

[0072] Acquire heterogeneous multimodal data and extract feature vectors for each modality. Label each feature vector with security attributes to generate a set of feature vectors with security labels, including:

[0073] Semantic space clustering analysis is performed on heterogeneous source multimodal data to identify the cluster center regions of heterogeneous source multimodal data in the semantic space as sensitive semantic regions;

[0074] Extract the feature vectors of each modality from heterogeneous source multimodal data, calculate the semantic distance between each feature vector and the center of the sensitive semantic region, and determine the semantic sensitivity value of each feature vector based on the semantic distance;

[0075] Each modal feature vector is mapped to a heterogeneous modal space through modal transformation to obtain a cross-modal mapped feature vector. The cross-modal mapped feature vector is then mapped back to the original modal space through inverse modal transformation to obtain a reconstructed feature vector. The feature difference degree between each modal feature vector and the reconstructed feature vector is calculated. Based on the feature difference degree, the cross-modal mapping stability value of each modal feature vector is determined.

[0076] Security attributes are labeled for each modality feature vector. The semantic sensitivity value and cross-modal mapping stability value are fused to calculate the security risk value of each modality feature vector. Security label levels are assigned to each modality feature vector according to the security risk value, and a set of feature vectors with security labels is generated.

[0077] After acquiring multimodal data from different data sources, the first step is to perform deep semantic space analysis on this heterogeneous multimodal data. Heterogeneous multimodal data may include different types of data such as text, images, audio, and video, originating from different acquisition devices, different business systems, or different network platforms. To identify potentially sensitive information, this multimodal data needs to be mapped to a unified semantic space for cluster analysis.

[0078] Specifically, text modal data is encoded into semantic vectors using a pre-trained language model, image modal data is processed using a convolutional neural network to extract visual semantic features, and audio modal data is processed using an acoustic model to extract audio semantic features. These semantic features from different modalities are mapped to the same high-dimensional semantic space, typically between 512 and 2048 dimensions. Within this semantic space, density clustering algorithms are used to cluster all data points, identifying densely distributed data regions. These dense regions often correspond to specific semantic themes, such as identity information involving personal privacy, financial data involving trade secrets, and sensitive content involving security control.

[0079] By analyzing historical labeled data and domain knowledge bases, we can predefine which semantic topics belong to sensitive categories. Matching the clustering results with the sensitive semantic topics identifies the cluster centers that highly overlap with the sensitive topics; these regions are the sensitive semantic regions. Each sensitive semantic region has a central coordinate in a high-dimensional semantic space, which represents the typical semantic representation of that sensitive topic.

[0080] After identifying the sensitive semantic regions, feature extraction was performed on the heterogeneous multimodal data. For the text modality, word embedding combined with a context encoder was used to extract text feature vectors; for the image modality, a deep convolutional network was used to extract multi-level visual features of the image; for the audio modality, Mel-spectrum analysis combined with a recurrent neural network was used to extract audio features; and for the video modality, spatiotemporal and motion features were extracted simultaneously. The feature vector dimension for each modality depends on the modality characteristics; typically, text feature vectors are 768-dimensional, image feature vectors are 2048-dimensional, and audio feature vectors are 512-dimensional.

[0081] After extracting the feature vectors for each modality, the semantic distance between each feature vector and the center of each sensitive semantic region is calculated. The semantic distance is measured using cosine distance or Euclidean distance. For each feature vector and the center of a sensitive semantic region, the cosine similarity between them is calculated, and then the similarity is converted into a distance metric. The minimum distance between the feature vector and all centers of sensitive semantic regions is taken as the sensitive distance of that feature vector. The smaller the distance, the closer the feature vector is to the sensitive content semantically.

[0082] Based on sensitive distance, distance values ​​are converted into semantic sensitivity values ​​through a nonlinear mapping function. The mapping function employs an exponential decay form, ensuring that feature vectors closer to the sensitive region receive higher sensitivity values. Specifically, the semantic sensitivity value ranges from 0 to 1; when the feature vector is exactly at the center of the sensitive semantic region, the sensitivity value is close to 1; when the feature vector is far from all sensitive semantic regions, the sensitivity value is close to 0. This continuous sensitivity measurement method can more finely characterize the sensitivity of feature vectors compared to binary classification.

[0083] After obtaining the semantic sensitivity values, it is also necessary to evaluate the stability of each modality feature vector during the cross-modal mapping process. Cross-modal mapping refers to the process of transforming feature vectors from one modality to another modality space, and this transformation is crucial for multimodal data fusion. However, some feature vectors may experience significant information loss or semantic drift during cross-modal mapping, and this instability can affect the accuracy of subsequent security identification.

[0084] To evaluate the stability of cross-modal mapping, the feature vectors of each modality are first mapped to a heterogeneous modal space through a modality transformation network. The modality transformation network is a multi-layer neural network whose input is the source modality feature vector and its output is the mapped feature vector in the target modality space. For example, it maps text feature vectors to an image feature space, or image feature vectors to an audio feature space. This cross-modal mapping can capture the semantic relationships between different modalities.

[0085] For each modal feature vector, it is sequentially mapped to all other modal spaces, resulting in multiple cross-modal mapped feature vectors. These cross-modal mapped feature vectors are then mapped back to the original modal space through the corresponding inverse modal transformation network (IMT), yielding reconstructed feature vectors. The structure of the IMT is symmetrical to that of the forward modal transformation network; its function is to restore the feature vectors of the target modal space back to the source modal space.

[0086] Calculate the feature difference between the original modality feature vector and the reconstructed feature vector. The feature difference is measured using mean squared error or cosine distance. If a feature vector can be reconstructed back to its original features well after cross-modal mapping and inverse mapping, it indicates that the feature vector maintains high information integrity and semantic consistency during cross-modal transformation and has high cross-modal mapping stability. Conversely, if the reconstruction error is large, it indicates that the feature vector has suffered significant information loss or semantic distortion during cross-modal transformation, and its cross-modal mapping stability is low.

[0087] The reconstruction errors of all heterogeneous modal spaces are weighted and averaged to obtain the comprehensive feature difference of the feature vector. Based on the feature difference, the cross-modal mapping stability value is calculated using the inverse mapping function. The stability value also ranges from 0 to 1; the smaller the feature difference, the higher the stability value. The stability value reflects the reliability of the feature vector in a multimodal fusion scenario, and feature vectors with high stability are more suitable for cross-modal correlation analysis and fusion processing.

[0088] After obtaining the semantic sensitivity and cross-modal mapping stability values, security attributes are labeled for each modality feature vector. The core of security attribute labeling is calculating the security risk value for each feature vector. The security risk value comprehensively considers two dimensions: the semantic sensitivity and cross-modal mapping stability of the feature vector. Feature vectors with high semantic sensitivity are more likely to contain sensitive information, while feature vectors with low cross-modal mapping stability are prone to unpredictable behavior in multimodal processing; both increase security risk.

[0089] Security risk values ​​are obtained through fusion calculations, employing either weighted summation or nonlinear combination. A common fusion method is to weight and sum the semantic sensitivity value with the cross-modal mapping instability value (i.e., 1 minus the stability value). The weighting coefficients are adjusted based on the specific application scenario and security strategy; typically, the weight for semantic sensitivity is set to 0.6 to 0.8, and the weight for cross-modal mapping instability is set to 0.2 to 0.4. This weighting allocation reflects the dominant role of semantic sensitivity in security risk assessment while also considering the impact of cross-modal processing stability.

[0090] Based on the calculated security risk values, security level labels are assigned to each modal feature vector. Security level labels are typically divided into multiple levels, such as low risk, medium risk, high risk, and extremely high risk. By setting threshold ranges, continuous security risk values ​​are mapped to discrete security levels. For example, feature vectors with security risk values ​​between 0 and 0.25 are labeled as low risk, those between 0.25 and 0.5 as medium risk, those between 0.5 and 0.75 as high risk, and those between 0.75 and 1 as extremely high risk.

[0091] After each feature vector is assigned a security label level, it is also appended with detailed security attribute metadata, including sensitive semantic categories, risk source modalities, sensitivity values, and stability values. This metadata provides a fine-grained basis for subsequent security control and risk propagation analysis. Ultimately, all feature vectors labeled with security attributes form a security-labeled feature vector set. Each element in this set carries complete security risk information, laying the foundation for subsequent cross-modal association graph construction and risk propagation analysis.

[0092] The feature vector set with security labels not only retains the original modal feature information, but also adds semantic annotations for security dimensions. This enables multimodal data processing systems to identify and mark potential security risk points during the feature extraction stage, thereby implementing targeted security control strategies in subsequent data fusion, correlation analysis and classification processes, effectively preventing the leakage of sensitive information and the spread of risks.

[0093] Constructing a cross-modal association graph based on the semantic similarity between feature vectors of different modalities in a feature vector set with security labels includes:

[0094] Extract the security label level information of each modal feature vector from the feature vector set with security labels, and perform security sensitivity weighting on each modal feature vector according to the security label level information, and assign a security sensitivity weight coefficient to each modal feature vector;

[0095] Semantic encoding projection is performed on each modality feature vector, and each modality feature vector is projected onto a unified semantic space to obtain a unified semantic representation vector. The vector cosine similarity between the unified semantic representation vectors of different modality feature vectors in the unified semantic space is calculated to obtain the initial semantic similarity between different modality feature vectors.

[0096] Security modulation of the initial semantic similarity is performed based on the security sensitivity weight coefficient. The security sensitivity weight coefficients corresponding to different modal feature vectors are multiplied with the initial semantic similarity to obtain the security-modulated semantic similarity.

[0097] Each modal feature vector in the set of feature vectors with security labels is used as a node in the cross-modal association graph. Edge connections are established between different modal feature vector pairs whose semantic similarity after security modulation exceeds a preset similarity threshold. The semantic similarity after security modulation is used as the weight value of the edge connection to generate the cross-modal association graph.

[0098] After obtaining the set of feature vectors with security labels, it is necessary to construct a cross-modal association graph that reflects the semantic relationships between data from different modalities. This association graph not only needs to capture cross-modal semantic similarity, but also needs to incorporate security attribute information so that subsequent risk analysis can accurately identify the propagation path of sensitive information.

[0099] First, the security label level information of each modal feature vector in the set of feature vectors with security labels is extracted. Security label level information typically includes sensitivity level labels, such as public, internal, confidential, and top secret classifications, or specific security category labels, such as personal privacy, trade secret, and technical secret. For different security label levels, corresponding security sensitivity weight coefficients are assigned to each modal feature vector. The security sensitivity weight coefficient reflects the sensitivity of the information carried by the feature vector; the higher the sensitivity level, the larger the weight coefficient. Specifically, security label levels can be mapped to numerical weight coefficients according to a predefined security level mapping table. For example, for a public-level feature vector, the security sensitivity weight coefficient can be set to 1.0; for an internal-level feature vector, the weight coefficient can be set to 1.5; for a confidential-level feature vector, the weight coefficient can be set to 2.5; and for a top-secret feature vector, the weight coefficient can be set to 4.0. This mapping relationship can be adjusted according to the security strategy in the actual application scenario. When a feature vector has multiple security labels, the weight coefficient corresponding to the highest sensitivity level is taken, or a weighted average of multiple weight coefficients is calculated. In this way, each modal feature vector is assigned a quantized security sensitivity weight coefficient, which will play a modulating role in subsequent semantic similarity calculations.

[0100] Next, semantic encoding projection is performed on the feature vectors of each modality. Since feature vectors of different modalities typically reside in their own independent feature spaces, their dimensions, distribution characteristics, and semantic representations differ significantly, making direct cross-modal semantic similarity calculation impossible. Therefore, semantic encoding projection is needed to map the feature vectors of each modality to a unified semantic space. Semantic encoding projection can be implemented using pre-trained cross-modal encoders, such as multimodal coding networks trained based on contrastive learning, or cross-modal mapping networks based on attention mechanisms. Specifically, text modality feature vectors are projected through a text encoder, image modality feature vectors through an image encoder, and audio modality feature vectors through an audio encoder. The output dimensions of each encoder are kept consistent, resulting in representation vectors located in a unified semantic space. The dimensions are typically set to 512 or 1024 to ensure sufficient semantic expressive power. In this unified semantic space, feature vectors of different modalities are comparable; semantically similar feature vectors are closer together, while semantically significantly different feature vectors are farther apart.

[0101] After obtaining the unified semantic representation vector, the semantic similarity between feature vectors from different modalities is calculated. For any two feature vectors from different modalities, their corresponding unified semantic representation vectors are obtained. and We calculate the cosine similarity between the two vectors as the initial semantic similarity. The formula for calculating cosine similarity is: ,in This represents the vector dot product operation. This represents the Euclidean norm of the vectors. Cosine similarity ranges from -1 to 1; a value closer to 1 indicates that the two vectors are in the same direction and have higher semantic similarity; a value closer to -1 indicates that they are in opposite directions and have lower semantic similarity. In practical applications, usually only positive similarity is considered. In this case, by traversing all different modal feature vector pairs in the set of feature vectors with security labels, an initial semantic similarity matrix can be obtained, which records the semantic association strength between any two different modal feature vectors.

[0102] However, the initial semantic similarity only reflects semantic-level correlation and does not consider the impact of security attributes on the correlation. In practical applications, the correlation between feature vectors involving sensitive information needs to be strengthened so that subsequent risk analysis can more accurately identify the propagation path of sensitive information. Therefore, it is necessary to perform security modulation on the initial semantic similarity based on a security sensitivity weight coefficient. Specifically, for feature vector pairs... Obtain their respective security sensitivity weight coefficients. and The two weight coefficients are multiplied by the initial semantic similarity to obtain the securely modulated semantic similarity. This modulation method significantly amplifies the semantic similarity of two feature vectors with high security sensitivity, resulting in stronger edge connections in the subsequently constructed cross-modal association graph. Conversely, when two feature vectors have low security sensitivity, their modulated similarity is not excessively amplified, even if they are semantically similar. This security modulation mechanism ensures that the cross-modal association graph highlights the relationships between nodes with sensitive information, providing a foundation for subsequent risk propagation analysis.

[0103] After obtaining the securely modulated semantic similarity, a cross-modal association graph is constructed. Each modal feature vector in the set of feature vectors with security labels is used as a node in the cross-modal association graph. Each node carries not only the original feature vector information but also security label level information and security sensitivity weight coefficients. For any two feature vector nodes from different modalities, if their securely modulated semantic similarity exceeds a preset similarity threshold, an edge is established between these two nodes. The similarity threshold needs to be adjusted according to the actual application scenario, typically set to a value between 0.3 and 0.6. Setting the threshold too low will result in too many edge connections in the association graph, increasing computational complexity and introducing a large number of weak associations; setting the threshold too high will result in too few edge connections, losing important cross-modal association information. When establishing an edge connection, the securely modulated semantic similarity is used as the weight value of the edge. The edge weight reflects the association strength between two nodes; a larger weight indicates a stronger association and a higher probability of information propagation.

[0104] Through the above process, a weighted undirected graph structure for cross-modal association is generated. The node set of this graph corresponds to a set of feature vectors with security labels, the edge set consists of feature vector pairs that satisfy a similarity threshold, and the edge weights are determined by the security-modulated semantic similarity. This cross-modal association graph not only captures the semantic relationships between different modalities of data but also incorporates security attribute information, thus strengthening the relationships between sensitive information nodes. In subsequent risk analysis, sensitive source nodes and risk convergence nodes can be identified based on this association graph, and risk propagation paths can be analyzed, thereby achieving secure identification and intelligent classification processing of multimodal data.

[0105] Furthermore, during the construction of the cross-modal association graph, modality type identifiers can be introduced to label each node with its corresponding modality type, such as text, image, audio, or video. These modality type identifiers help identify the modal distribution characteristics of cross-modal associations in subsequent analysis. For example, they can identify which modalities a sensitive source node is primarily associated with, or which modalities of sensitive information a risk aggregation node aggregates. In this way, the cross-modal association graph becomes a comprehensive graph structure that includes semantic association information, security attribute information, and modality type information, providing a rich data foundation for subsequent multi-dimensional risk analysis.

[0106] In the cross-modal association graph, the feature vectors with security labels are used as sensitive source nodes. The neighborhood node set and modal distribution characteristics of each sensitive source node are calculated. Based on the neighborhood node set and modal distribution characteristics, the sensitive source influence indicators are determined, including:

[0107] In the cross-modal association graph, nodes with security-labeled feature vectors whose security label level exceeds the preset security level threshold are extracted as sensitive source nodes. For each sensitive source node, multi-hop graph expansion is performed along the edge connection in descending order of weight value to obtain the set of nodes that can be reached by each sensitive source node within a preset expansion step as the neighborhood node set.

[0108] Calculate the shortest path length between each node in the neighborhood node set and the sensitive source node, and use the reciprocal of the shortest path length as the distance decay factor. Assign the distance decay factor as the node weight to each node in the neighborhood node set.

[0109] The sum of node weights for each modality type in the neighborhood node set is counted, and the proportion of the sum of node weights for each modality type to the total weight of the neighborhood node set is calculated as the modality distribution feature. The information entropy of the modality distribution feature is then calculated to obtain the modality distribution uniformity value.

[0110] The total weight of the neighborhood node set is used as the weighted diffusion range value. The weighted diffusion range value is multiplied by the modal distribution uniformity value to obtain the sensitive source influence index.

[0111] After constructing the cross-modal association graph, it is necessary to assess the influence of sensitive source nodes in the graph to quantify their potential risk diffusion ability to surrounding nodes. First, sensitive source nodes are identified from the cross-modal association graph. By comparing the security label level of each feature vector node with a security identifier with a preset security level threshold, nodes with security label levels exceeding the threshold are selected as sensitive source nodes. The preset security level threshold can be set according to the actual application scenario. For example, it can be set to level 7 (out of 10) in financial data processing scenarios and level 8 in medical data processing scenarios, ensuring that only nodes with truly high security risks are marked as sensitive sources.

[0112] For each identified sensitive source node, its influence range, i.e., the set of neighboring nodes, needs to be determined. A multi-hop graph expansion strategy is adopted, starting from the sensitive source node and expanding along the edges in the cross-modal association graph. During the expansion process, edges with higher weight values ​​are prioritized for traversal because higher weight edges represent stronger semantic associations, meaning a greater possibility of risk propagation. In specific implementation, all outgoing edges of each sensitive source node are sorted in descending order of weight value, and neighboring nodes are visited sequentially. In the first hop expansion, all nodes directly connected to the sensitive source node are directly obtained; in the second hop expansion, the nodes obtained in the first hop continue to expand outward, obtaining nodes two steps away from the sensitive source node; and so on, until the preset expansion steps are reached. The preset expansion steps are usually set to 3 to 5 steps, which can cover a sufficient influence range while avoiding excessive computational complexity. For example, when the preset expansion steps are 3, for a certain sensitive source node, 12 directly adjacent nodes are obtained in the first hop, 34 nodes are added in the second hop, and 58 nodes are added in the third hop, and finally the neighborhood node set of the sensitive source node contains 104 nodes.

[0113] After obtaining the neighborhood node set, each node in the set needs to be assigned a weight to reflect its affinity with the sensitive source node. The shortest path length between each node in the neighborhood node set and the sensitive source node is calculated using a breadth-first search algorithm or Dijkstra's algorithm from graph theory. The shortest path length represents the minimum number of edges required to reach a neighboring node from the sensitive source node. The reciprocal of the shortest path length is used as the distance decay factor, calculated using the following formula: ,in Indicates the first The distance attenuation factor is calculated as the shortest path length between a neighboring node and the sensitive source node. For example, if the shortest path length between a neighboring node and the sensitive source node is 2, then its distance attenuation factor is 0.5; if the shortest path length is 3, then the distance attenuation factor is 0.333. This design conforms to the attenuation law of risk propagation; the farther a node is from the sensitive source, the less affected it is. The calculated distance attenuation factor is used as a node weight and assigned to each node in the neighboring node set, forming a weighted neighboring node set.

[0114] Based on the weighted neighborhood node set, the modal distribution characteristics are analyzed. The sum of node weights for each modal type in the neighborhood node set is calculated. Assuming the cross-modal association graph involves three modalities—text, image, and audio—the sum of weights for all neighborhood nodes belonging to the text modality is calculated separately. The sum of the weights of all neighboring nodes belonging to the image modality The sum of the weights of all neighboring nodes belonging to the audio modality Calculate the total weight of the neighborhood node set. The proportion of the sum of the weights of nodes of each modality type to the total weight of the neighboring node set is used as the modality distribution feature, i.e., the text modality distribution proportion. Image modal distribution ratio Audio modal distribution ratio These proportions reflect the distribution of the influence of sensitive source nodes across different modes.

[0115] To quantify the uniformity of modal distribution, information entropy is calculated for the modal distribution characteristics. Information entropy measures the uncertainty or uniformity of the distribution. When the proportions of each modality are similar, the information entropy is large, indicating that the influence is evenly distributed; when a certain mode dominates, the information entropy is small, indicating that the influence is concentrated in that specific mode.

[0116] The formula for calculating the modal distribution uniformity value is as follows: ,in Indicates the total number of modal types. Indicates the first The distribution ratio of different modes. Taking three modes as an example, When the distribution ratio of all three modes is 0.333, the information entropy reaches its maximum value of about 1.585, indicating that the influence is completely uniformly distributed; when the ratio of one mode is 1 and the others are 0, the information entropy is 0, indicating that the influence is completely concentrated.

[0117] The total weight of the neighboring node set is used as the weighted diffusion range value. This value directly reflects the size of the influence range of the sensitive source node. The larger the weighted diffusion range value, the more nodes the sensitive source node can influence and the closer their connections. The sensitive source influence index is obtained by multiplying the weighted diffusion range value by the modal distribution uniformity value. This indicator comprehensively considers both the scope of influence and the uniformity of influence distribution, identifying both sensitive sources with a wide scope of influence and those with strong cross-modal influence capabilities. For example, a sensitive source node with a weighted diffusion range of 45.6 and a modal distribution uniformity of 1.42 has a sensitive source influence index of 64.75. Another sensitive source node has a weighted diffusion range of 52.3, but a modal distribution uniformity of only 0.68 (its influence is mainly concentrated in a single mode), resulting in a sensitive source influence index of 35.56. Although its diffusion range is larger, its overall influence is lower due to its weak cross-modal influence capability.

[0118] In practical applications, for large-scale cross-modal association graphs containing hundreds of sensitive source nodes, parallel computing strategies can be employed to accelerate the extraction of neighborhood node sets and the calculation of influence indicators. The computational tasks for different sensitive source nodes are distributed to different computing units, each independently completing multi-hop graph expansion, shortest path calculation, weight statistics, and information entropy calculation. Finally, the influence indicators of all sensitive source nodes are aggregated. For dynamically updated multimodal data streams, when new feature vectors with security labels are added to the cross-modal association graph, only local updates are needed for the newly added sensitive source nodes and the nodes within their influence range; the influence indicators of the entire graph do not need to be recalculated, significantly improving computational efficiency. Using the sensitive source influence indicators, all sensitive source nodes can be ranked, prioritizing nodes with high influence indicators, providing a quantitative basis for subsequent risk convergence analysis and risk propagation modeling.

[0119] Nodes associated with multiple sensitive source nodes in the cross-modal correlation graph are identified as risk convergence nodes. The security type of the sensitive source nodes associated with each risk convergence node is calculated, and the risk convergence coefficient is determined based on the security type, including:

[0120] Traverse all nodes in the cross-modal association graph, count the number of sensitive source nodes directly associated with each node, and identify nodes whose number of directly associated sensitive source nodes exceeds a preset aggregation threshold as risk aggregation nodes;

[0121] Extract the security identification level information of each sensitive source node directly associated with the risk aggregation node, classify each sensitive source node into security types based on the security identification level information, and obtain the security type set of the risk aggregation node;

[0122] Traverse the security types in the security type set, determine the conflict between any two security types according to the preset security type conflict rules, and construct a conflict topology graph by using the security type pairs with conflicting relationships as edges. The nodes of the conflict topology graph are the security types in the security type set.

[0123] Calculate the degree of each node in the conflict topology graph, where the degree is the number of edges connected to the current node. Mark the node with the largest degree as the core conflict node and obtain the security type corresponding to the core conflict node as the core conflict type.

[0124] The number of sensitive source nodes whose security type belongs to the core conflict type among the directly associated sensitive source nodes of the statistical risk aggregation node is used as the risk aggregation coefficient.

[0125] After constructing the cross-modal association graph and calculating the influence indicators of each sensitive source node, it is necessary to further identify risk convergence phenomena in the graph. Risk convergence nodes are special nodes that are simultaneously associated with multiple sensitive source nodes. These nodes may become key locations where different security risks intersect, and changes in their security status may trigger chain reactions.

[0126] For each node in the cross-modal association graph, traverse and scan it to count the number of sensitive source nodes directly connected to that node through edges. Specifically, for any node in the graph... First, check all adjacent nodes to determine if any of these adjacent nodes carry a security identifier. Then, increment the count of adjacent nodes carrying security identifiers to obtain the node count. Number of directly associated sensitive source nodes Set a preset convergence threshold. ,when At that time, the node Marked as a risk aggregation node. This threshold is usually set according to the security control requirements of the actual application scenario. For example, in a financial data processing scenario, it may be set to 3, indicating that a node that is simultaneously associated with more than 3 sensitive sources needs to be closely monitored.

[0127] After identifying all risk aggregation nodes, the security identification level information of the associated sensitive source nodes is extracted for each risk aggregation node. This security identification level information is usually already labeled on the feature vector during the data acquisition phase, such as "Confidential," "Sensitive," and "Restricted." Based on this security identification level information, sensitive source nodes are classified into different security types. The specific classification method can be based on the semantic meaning of the identification level; for example, "Confidential" and "Top Secret" can be classified as highly sensitive, and "Sensitive" and "Internal" as moderately sensitive. The "restricted level" is classified as a low-sensitivity type. For risk convergence nodes... The security types of all the sensitive source nodes associated with it constitute a set.

[0128] After obtaining the set of security types, it is necessary to analyze the conflict relationships between different security types. Predefined security type conflict rules define which combinations of security types will generate security risks. For example, the rules might stipulate that "the simultaneous occurrence of highly sensitive and low-sensitive types within the same node's neighborhood constitutes a conflict," or "the simultaneous occurrence of types involving personal privacy and types involving trade secrets constitutes a conflict." The process involves iterating through all security type pairs in the set. The system determines whether a conflict exists between pairs of this type based on conflict rules. If a conflict exists, it is added to the conflict topology graph. and Consider two nodes and add an edge between them. (Conflict topology graph) The set of nodes is The edge set is a set of all safe type pairs that have conflicting relationships.

[0129] After constructing the conflict topology graph, calculate the degree of each node in the graph. degree Defined as the number of edges directly connected to a node, this reflects the frequency with which this security type conflicts with other security types. A higher degree indicates that the security type conflicts with more other types, potentially playing a more critical role in risk propagation. Traverse all nodes in the conflict topology graph, find the node with the highest degree, and mark it as the core conflict node. If multiple nodes have the same maximum degree, further selection can be based on the priority of security identifier levels, for example, prioritizing highly sensitive types as core conflict nodes. The security type corresponding to the core conflict node. This is the core conflict type, which dominates the security risk assessment of the current risk convergence node.

[0130] After identifying the core conflict types, statistical risk aggregation nodes are determined. Among directly associated sensitive source nodes, the security type belongs to The number of nodes. Assume a risk aggregation node. The set of directly associated sensitive source nodes is Iterate through each sensitive source node in the set and check if its security type is [missing information]. The number of sensitive source nodes that meet the conditions is accumulated. This quantity directly reflects the distribution density of core conflict types within the neighborhood of the current risk convergence node. A larger quantity indicates a more significant impact of the core conflict type on the node, and a higher degree of risk convergence. Therefore, [the following is a continuation of the previous sentence, but the context is unclear]. As a risk convergence node Risk Convergence Coefficient .

[0131] The calculation of the risk convergence coefficient fully considers the diversity and conflict of security types. By constructing a conflict topology graph, the core type with the greatest influence when multiple security types intersect can be identified, avoiding simply using the number of all sensitive source nodes as a risk measure, thus more accurately depicting the essential characteristics of risk convergence. For example, a risk convergence node may be associated with 5 sensitive source nodes, but if the security types of these 5 nodes are highly consistent and there is no conflict, its risk convergence coefficient may be low; conversely, if the associated sensitive source nodes have diverse security types and serious conflicts, even if the number is small, the risk convergence coefficient will be high.

[0132] In practical applications, the risk convergence coefficient can be combined with the sensitive source influence index to jointly construct a risk propagation model. The risk convergence coefficient focuses on describing the characteristics of nodes as risk convergence points, while the sensitive source influence index focuses on describing the ability of sensitive source nodes to spread risk outward. The combination of the two can comprehensively characterize the risk distribution pattern in the cross-modal association graph, providing a reliable quantitative basis for subsequent risk propagation matrix construction and feature fusion classification.

[0133] For cross-modal association graphs containing a large number of nodes, the identification of risk-converging nodes and the calculation of risk-converging coefficients can be improved through parallelization. Since the calculation of the risk-converging coefficient for each node is independent, the node set can be divided into multiple subsets and distributed to different computing units for parallel processing. The construction of conflict topology graphs can also adopt an incremental update approach. When a new sensitive source node is added or the security type changes, only the affected part is updated without rebuilding the entire graph structure.

[0134] A risk propagation matrix is ​​constructed based on the influence index of sensitive sources and the risk convergence coefficient. The risk propagation matrix is ​​then subjected to eigenvalue decomposition to identify the dominant risk propagation direction, generating a risk propagation vector including:

[0135] Traverse the propagation path from the sensitive source node to the risk convergence node in the cross-modal association graph, extract the modal types of adjacent nodes on the propagation path, determine whether there is a propagation blocking relationship between the modal types of adjacent nodes according to the preset modal compatibility rules, and count the number of adjacent node pairs with a propagation blocking relationship in the propagation path as the number of propagation blocking points.

[0136] A propagation path with zero propagation blockade points is marked as a valid propagation path;

[0137] Extract the sensitive source influence index of sensitive source nodes and the risk convergence coefficient of risk convergence nodes corresponding to effective propagation paths. For each node pair from a sensitive source node to each risk convergence node, count the number of effective propagation paths connecting the node pairs as the path connectivity.

[0138] A risk propagation matrix is ​​constructed based on the sensitive source influence index corresponding to the sensitive source node, the risk convergence coefficient corresponding to the risk convergence node, and the path connectivity of the node pair. The row index of the risk propagation matrix is ​​the sensitive source node, and the column index is the risk convergence node. For each node pair, the corresponding sensitive source influence index, risk convergence coefficient, and path connectivity are used as input parameters to calculate the matrix element value.

[0139] The risk propagation matrix is ​​decomposed into eigenvalues, and the eigenvector corresponding to the largest eigenvalue is used as the risk propagation vector.

[0140] After obtaining the sensitivity influence index of each sensitive source node and the risk convergence coefficient of each risk convergence node, it is necessary to further analyze the risk propagation mechanism in the cross-modal association graph. Risk propagation is not a simple direct connection between nodes, but rather spreads through multiple paths between nodes of different modalities. Due to the heterogeneity of different modal data, there may be natural propagation barriers between certain modal combinations. For example, the significant differences in semantic expression between text modal nodes and audio modal nodes may weaken or even block risk information during propagation. Therefore, before constructing the risk propagation matrix, it is necessary to identify which propagation paths in the cross-modal association graph are truly effective.

[0141] The process iterates through all propagation paths in the cross-modal association graph from sensitive source nodes to risk convergence nodes. For each propagation path, the modality types of adjacent nodes are extracted sequentially according to the node access order. For example, if a propagation path is "text node A → image node B → text node C → audio node D", then the adjacent node pairs include three modality type combinations: "text-image", "image-text", and "text-audio". The modality type of each pair of adjacent nodes is determined according to a preset modality compatibility rule. The modality compatibility rule is based on the semantic conversion difficulty and information fidelity between different modalities. Specifically, text modality and image modality have high compatibility due to mature image-text matching technology and do not constitute a propagation block; image modality and audio modality have low compatibility due to the lack of a direct semantic mapping mechanism and may constitute a propagation block; text modality and audio modality have high compatibility if speech recognition or speech synthesis scenarios are involved, otherwise, their compatibility is low. When the modality type combination of adjacent nodes is determined to have a propagation blocking relationship, the adjacent node pair is counted as a propagation blocking point. The number of propagation blocking points in the entire propagation path is obtained by counting the number of adjacent node pairs that have a propagation blocking relationship.

[0142] Propagation paths with zero propagation blocking points are marked as effective propagation paths. This means that all modality combinations of adjacent nodes on this path have good compatibility, and risk information can propagate smoothly along the path without interruption due to modality transition failure. Effective propagation paths constitute the actual channels for risk diffusion in the cross-modal association graph and are the foundation for subsequent risk propagation matrix construction. For paths with more than zero propagation blocking points, although there are theoretically connections from sensitive source nodes to risk convergence nodes, their role in actual risk propagation is weakened due to modality compatibility barriers, and therefore they are not included in the set of effective propagation paths.

[0143] After identifying all valid propagation paths, the sensitivity influence index of the sensitive source nodes and the risk convergence coefficient of the risk convergence nodes corresponding to these paths are extracted. For each node pair formed by a sensitive source node and a risk convergence node in the cross-modal association graph, the number of valid propagation paths connecting the node pair is counted. This number reflects the richness of channels through which risk information from the sensitive source node reaches the risk convergence node, and is called path connectivity. The higher the path connectivity, the more independent and valid propagation channels exist between the sensitive source node and the risk convergence node, and the stronger the robustness of risk propagation. Even if a path is interrupted due to the failure of a local node, the risk can still continue to propagate through other paths. Node pairs with zero path connectivity indicate that the risk from the sensitive source node cannot reach the risk convergence node through valid paths. The element value corresponding to such node pairs in the risk propagation matrix will be set to zero or a minimum value.

[0144] A risk propagation matrix is ​​constructed based on the sensitive source influence index corresponding to sensitive source nodes, the risk convergence coefficient corresponding to risk convergence nodes, and the path connectivity of node pairs. The row indices of this matrix correspond to all sensitive source nodes in the cross-modal association graph, and the column indices correspond to all risk convergence nodes. Assume there are a total of [number missing] sensitive source nodes in the cross-modal association graph. Sensitive source nodes and If there are multiple risk convergence nodes, then the risk propagation matrix... The dimension is For each element in the matrix Its corresponding sensitive source node Risk convergence node The resulting node pairs. The calculation of matrix element values ​​comprehensively considers three factors: sensitive source nodes. Sensitive source influence indicators This reflects the ability of the sensitive source node to spread risk outwards; risk convergence node. Risk Convergence Coefficient This reflects the degree to which the risk aggregation node gathers multi-source risks, and the path connectivity. This reflects the number of propagation channels between two nodes. The matrix element values ​​are obtained through the formula... The calculation yielded the results. The logarithmic transformation of path connectivity was used to prevent excessively large matrix element values ​​from masking the influence of other factors when the number of paths was too high, while preserving the positive promoting effect of path connectivity on risk propagation intensity. When path connectivity... When it is zero, Corresponding matrix element values This indicates that there is no effective risk propagation between the node pairs.

[0145] Completed risk propagation matrix The distribution of risk flow intensity from sensitive source nodes to risk convergence nodes in the cross-modal correlation graph is characterized. To identify the dominant direction of risk propagation, eigenvalue decomposition is performed on the risk propagation matrix. Eigenvalue decomposition transforms the matrix... Decomposed into In the form of, It is a diagonal matrix, and the diagonal elements are eigenvalues. Sort in descending order; and These are the left and right eigenvector matrices, respectively. The largest eigenvalue... The corresponding eigenvector represents the most concentrated propagation pattern in the risk propagation matrix, i.e., the dominant risk propagation direction. Obtain the largest eigenvalue. The corresponding right eigenvector This serves as a risk propagation vector. The dimension of this vector is... Each component corresponds to a risk convergence node, and the magnitude of the component value reflects the importance of that risk convergence node in the direction of dominant risk propagation. Risk convergence nodes with larger component values ​​in the risk propagation vector are the main target nodes for risk propagation. These nodes are located in key downstream positions in the risk flow in the cross-modal association graph, and the monitoring and protection of these nodes should be prioritized.

[0146] The risk propagation vector not only reveals the relative importance of risk convergence nodes but also implies the overall pattern of risk transmission from sensitive source nodes to these nodes through effective propagation paths. By analyzing the correlation between the risk propagation vector and each column of the risk propagation matrix, we can further identify which sensitive source nodes contribute the most to the dominant risk propagation direction. This provides a basis for subsequent risk tracing and targeted prevention and control. Simultaneously, as a global representation of the risk propagation characteristics of the cross-modal association graph, the risk propagation vector will be used in subsequent steps to calculate the directional decay factor of each edge, thereby adjusting the edge weights of the cross-modal association graph and making the graph structure more accurately reflect the actual dynamics of risk propagation.

[0147] like Figure 2 The diagram shows the flowchart for cross-modal association graph node classification processing in this embodiment.

[0148] The directional decay factor of each edge in the cross-modal association graph is calculated based on the risk propagation vector. The edge weights are adjusted based on the directional decay factor to generate an adjusted cross-modal association graph. Feature fusion and classification processing are then performed based on the adjusted cross-modal association graph, including:

[0149] Extract the vector component values ​​corresponding to each node in the risk propagation vector. For each edge in the cross-modal association graph, calculate the difference between the vector component values ​​of the starting node and the ending node of the edge. Calculate the directional decay factor of the edge based on the difference.

[0150] The original weight values ​​of each edge in the cross-modal association graph are attenuated based on the directional attenuation factor to obtain the adjusted edge weight values, and the adjusted cross-modal association graph is generated.

[0151] Based on the adjusted cross-modal association graph, the feature vectors of the neighboring nodes of each node are weighted and aggregated according to the adjusted edge weight values ​​to obtain the fused feature vector.

[0152] Construct a category prototype library, calculate the similarity between the fused feature vector and the feature vectors of each category prototype in the category prototype library, and take the category with the highest similarity as the candidate classification result;

[0153] Extract the candidate classification results of the node in the adjusted cross-modal association graph, count the class frequency of the candidate classification results of the neighboring nodes, take the class with the highest class frequency as the dominant class of the neighborhood, and determine whether the candidate classification result of the node is consistent with the dominant class of the neighborhood. If they are consistent, the candidate classification result is taken as the final classification result. If they are inconsistent, the candidate classification result is corrected based on the adjusted edge weight value to obtain the final classification result.

[0154] In obtaining the risk propagation vector Next, this vector needs to be used to adjust the edge weights in the cross-modal association graph to reflect the directional characteristics of risk propagation. The dimension of the risk propagation vector is the same as the total number of nodes in the cross-modal association graph, and each node corresponds to a component value in the vector. The vector component values ​​corresponding to each node in the risk propagation vector are extracted, and the node is denoted as... The corresponding vector component values ​​are ,node The corresponding vector component values ​​are For connected nodes in a cross-modal association graph and nodes edge , will node Considered as the starting node, node Treating the endpoint as the starting node, calculate the difference between the vector component values ​​of the starting node and the endpoint. The difference reflects the directional strength of risk propagation along that edge. When the vector component value of the endpoint node is significantly higher than that of the starting node, it indicates a stronger trend of risk propagation along that direction; conversely, it indicates a weaker contribution of that edge to risk propagation. Difference The calculation uses the normalized difference form, specifically as follows: ,in To prevent division by zero of tiny positive numbers, the value is usually taken as 0.0001. When the value is positive, it indicates that the risk follows the node. To the node Propagate in the direction of, when When the value is negative, it indicates that the direction is opposite to the direction of the dominant risk propagation.

[0155] Based on difference Calculate edges Directional attenuation factor The directional decay factor is used to quantize the adjustment magnitude of the edge weights, and its calculation formula is as follows: ,in This is a parameter for controlling the attenuation intensity, typically ranging from 1.0 to 3.0. When the difference... When it is a positive value, Zero, directional attenuation factor An edge equal to 1 indicates that it aligns with the direction of risk propagation and does not undergo attenuation. When the difference... When it is negative, equal The absolute value of the directional attenuation factor A value less than 1 indicates that the edge propagates in the opposite direction to the risk propagation, requiring weight attenuation. Attenuation strength control parameter. The larger the value, the more pronounced the attenuation effect. In practical applications, the value can be adjusted according to the sensitivity of different security scenarios to the directionality of risk propagation. For scenarios with high security requirements, the value of can be appropriately increased. This is to strengthen directional constraints.

[0156] After obtaining the directional attenuation factor for each edge, attenuation calculations are performed on the original weight values ​​of each edge in the cross-modal correlation graph. (The edge is then denoteed.) The original weight values ​​are The original weight values ​​are typically obtained by securely modulating semantic similarity. Adjusted edge weight values. It is obtained by multiplying the original weight value by the directional attenuation factor, i.e. The decay operation is performed on all edges in the cross-modal association graph to generate an adjusted cross-modal association graph. The adjusted cross-modal association graph retains the original set of nodes and edge connections, but the edge weights are differentiated according to the direction of risk propagation. Edges consistent with the dominant risk propagation direction retain their original weights, while the weights of edges opposite to the dominant risk propagation direction are decayed, thereby reducing the impact of reverse risk propagation paths in subsequent feature fusion processes.

[0157] Feature fusion is performed based on the adjusted cross-modal association graph. For each node in the cross-modal association graph, the set of its neighboring nodes is extracted. The set of neighboring nodes includes all nodes directly connected to that node. Let the node to be fused be denoted as . Its neighborhood node set is The set of neighboring nodes The neighboring nodes are denoted as ,node With neighboring nodes The adjusted weight values ​​between them are Extracting neighboring nodes corresponding feature vector This feature vector is the modal feature vector obtained by the node in the initial feature extraction stage. The feature vectors of neighboring nodes are then weighted and aggregated, with the weighting coefficients being the adjusted edge weights. Node fused feature vector The calculation formula is This weighted aggregation process achieves cross-modal feature fusion. Since the edge weights have been adjusted according to the risk propagation direction, the fused feature vector can more accurately reflect the semantic representation of the node under security constraints, reducing the interference of the risk propagation path on feature fusion.

[0158] A category prototype library is constructed for classification processing. The library contains prototype feature vectors for each target category, with one prototype feature vector corresponding to each category. These prototype feature vectors can be obtained by calculating cluster centers from the fused feature vectors of samples from each category in the training dataset, or by using predefined standard feature templates. Let the total number of categories in the prototype library be... The first category, the The prototype feature vectors of each category are denoted as follows: For nodes fused feature vector The similarity between the node and the feature vector of each category prototype in the category prototype library is calculated. The similarity calculation uses cosine similarity as the metric. The fusion feature vector and the first Similarity of prototype feature vectors of each category The calculation formula is Iterate through all categories in the category prototype library, find the category with the highest similarity, and set that category as a node. Candidate classification results ,Right now .

[0159] To improve the robustness of classification results, a neighborhood consistency verification mechanism is introduced. Node extraction. Candidate classification results for neighboring nodes in the adjusted cross-modal association graph. Traverse the set of neighboring nodes. For each neighboring node in the dataset, obtain the candidate classification result for each neighboring node. Count the class frequency of the candidate classification results for each neighboring node, and denote it as the i-th... The number of times each category appears in the candidate classification results of neighboring nodes is: The category with the highest frequency is selected as the dominant category in the neighborhood. ,Right now The dominant neighborhood category reflects the node's... The mainstream classification tendency of surrounding nodes, when node When the candidate classification result is consistent with the dominant category in the neighborhood, it indicates that the classification result of the node has good consistency with its neighboring nodes, and the classification result has high credibility.

[0160] Determine the node Candidate classification results Is it related to the dominant category in the neighborhood? Consistent. If If the candidate classification result passes the neighborhood consistency check, then the candidate classification result is considered to have passed the neighborhood consistency check. As a node Final classification results .like If the candidate classification result conflicts with the dominant category in the neighborhood, then neighborhood correction is required. The neighborhood correction process involves weighted voting on the candidate classification results based on the adjusted edge weights. This applies to the set of neighborhood nodes. Each neighboring node in Extract its candidate classification results and nodes Adjusting the weight values ​​between Weighted voting is conducted for each category, and the [number]th [class] is [number]. Weighted voting scores for each category The calculation formula is Also consider nodes The confidence level of the node's own candidate classification result will determine the node's confidence level. Similarity to its most similar class prototype This weight, used as the voting weight for its own candidate classification result, is added to the weighted voting score of the corresponding category. The corrected weighted voting score. for ,in This is the self-confidence weighting coefficient, typically ranging from 0.5 to 2.0. This is an indicator function; it takes a value of 1 when the condition within the parentheses is true, and 0 otherwise. The category with the highest weighted voting score after correction is selected as the node. Final classification results ,Right now The neighborhood correction mechanism can effectively correct classification errors caused by local feature anomalies or noise interference, thereby improving the overall consistency and accuracy of classification results.

[0161] The above feature fusion and classification process is performed on all nodes in the cross-modal association graph to obtain the final classification result for each node. The final classification result comprehensively considers the similarity between the node's own fused feature vector and the category prototype, as well as the classification tendency of the node's neighborhood, and achieves safe identification and intelligent classification processing of multimodal data under the constraint of risk propagation directionality.

[0162] A second aspect of this invention provides a secure identification and intelligent classification processing system for multimodal data, comprising:

[0163] The multimodal labeling unit is used to acquire heterogeneous source multimodal data and extract feature vectors of each modality, label each feature vector with security attributes, and generate a set of feature vectors with security labels.

[0164] Cross-modal building blocks are used to construct cross-modal association graphs based on the semantic similarity between feature vectors of different modalities in a set of feature vectors with security labels;

[0165] Sensitive source refers to the unit, which is used in the cross-modal association graph to use feature vectors with security labels as sensitive source nodes, calculate the neighborhood node set and modal distribution characteristics of each sensitive source node, and determine the sensitive source influence index based on the neighborhood node set and modal distribution characteristics;

[0166] The risk aggregation unit is used to identify nodes that are associated with multiple sensitive source nodes in the cross-modal association graph as risk aggregation nodes, count the security type of the sensitive source nodes associated with each risk aggregation node, and determine the risk aggregation coefficient based on the security type.

[0167] The risk propagation unit is used to construct a risk propagation matrix based on the influence index of sensitive sources and the risk convergence coefficient, perform eigenvalue decomposition on the risk propagation matrix to identify the dominant risk propagation direction, and generate a risk propagation vector.

[0168] The fusion classification unit is used to calculate the directional decay factor of each edge in the cross-modal association graph based on the risk propagation vector, adjust the edge weights based on the directional decay factor, generate an adjusted cross-modal association graph, perform feature fusion based on the adjusted cross-modal association graph, and perform classification processing.

[0169] A third aspect of the present invention provides an electronic device, comprising:

[0170] processor;

[0171] Memory used to store processor-executable instructions;

[0172] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0173] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0174] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for secure identification and intelligent classification of multimodal data, characterized in that, include: Acquire heterogeneous source multimodal data and extract feature vectors for each modality. Label each feature vector with security attributes to generate a set of feature vectors with security labels. A cross-modal association graph is constructed based on the semantic similarity between feature vectors of different modalities in a feature vector set with security labels; In the cross-modal association graph, the feature vector with security label is used as the sensitive source node. The set of neighborhood nodes and modal distribution characteristics of each sensitive source node are calculated. The influence index of the sensitive source is determined based on the set of neighborhood nodes and modal distribution characteristics. Identify nodes that are associated with multiple sensitive source nodes in the cross-modal association graph as risk aggregation nodes, count the security type of the sensitive source nodes associated with each risk aggregation node, and determine the risk aggregation coefficient based on the security type; A risk propagation matrix is ​​constructed based on the influence index of sensitive sources and the risk convergence coefficient. The risk propagation matrix is ​​then decomposed by eigenvalue to identify the dominant risk propagation direction and generate a risk propagation vector. The directional decay factor of each edge in the cross-modal association graph is calculated based on the risk propagation vector. The edge weights are adjusted based on the directional decay factor to generate an adjusted cross-modal association graph. Feature fusion and classification are then performed based on the adjusted cross-modal association graph. Acquire heterogeneous source multimodal data and extract feature vectors for each modality. Label each feature vector with security attributes to generate a set of feature vectors with security labels, including: Semantic space clustering analysis is performed on heterogeneous source multimodal data to identify the cluster center regions of heterogeneous source multimodal data in semantic space as sensitive semantic regions; Extract the feature vectors of each modality from heterogeneous source multimodal data, calculate the semantic distance between each feature vector and the center of the sensitive semantic region, and determine the semantic sensitivity value of each feature vector based on the semantic distance; Each modal feature vector is mapped to a heterogeneous modal space through modal transformation to obtain a cross-modal mapped feature vector. The cross-modal mapped feature vector is then mapped back to the original modal space through inverse modal transformation to obtain a reconstructed feature vector. The feature difference degree between each modal feature vector and the reconstructed feature vector is calculated. Based on the feature difference degree, the cross-modal mapping stability value of each modal feature vector is determined. Security attributes are labeled for each modality feature vector. The semantic sensitivity value and cross-modal mapping stability value are fused to calculate the security risk value of each modality feature vector. Security label levels are assigned to each modality feature vector according to the security risk value, and a set of feature vectors with security labels is generated. The directional decay factor of each edge in the cross-modal association graph is calculated based on the risk propagation vector. The edge weights are adjusted based on the directional decay factor to generate an adjusted cross-modal association graph. Feature fusion and classification processing are then performed based on the adjusted cross-modal association graph, including: Extract the vector component values ​​corresponding to each node in the risk propagation vector. For each edge in the cross-modal association graph, calculate the difference between the vector component values ​​of the starting node and the ending node of the edge. Calculate the directional decay factor of the edge based on the difference. The original weight values ​​of each edge in the cross-modal association graph are attenuated based on the directional attenuation factor to obtain the adjusted edge weight values, and the adjusted cross-modal association graph is generated. Based on the adjusted cross-modal association graph, the feature vectors of the neighboring nodes of each node are weighted and aggregated according to the adjusted edge weight values ​​to obtain the fused feature vector. Construct a category prototype library, calculate the similarity between the fused feature vector and the feature vectors of each category prototype in the category prototype library, and take the category with the highest similarity as the candidate classification result; Extract the candidate classification results of the node in the adjusted cross-modal association graph, count the class frequency of the candidate classification results of the neighboring nodes, take the class with the highest class frequency as the dominant class of the neighborhood, and determine whether the candidate classification result of the node is consistent with the dominant class of the neighborhood. If they are consistent, the candidate classification result is taken as the final classification result. If they are inconsistent, the candidate classification result is corrected based on the adjusted edge weight value to obtain the final classification result.

2. The method according to claim 1, characterized in that, Constructing a cross-modal association graph based on the semantic similarity between feature vectors of different modalities in a feature vector set with security labels includes: Extract the security label level information of each modal feature vector from the feature vector set with security labels, and perform security sensitivity weighting on each modal feature vector according to the security label level information, and assign a security sensitivity weight coefficient to each modal feature vector; Semantic encoding projection is performed on each modality feature vector, and each modality feature vector is projected onto a unified semantic space to obtain a unified semantic representation vector. The vector cosine similarity between the unified semantic representation vectors of different modality feature vectors in the unified semantic space is calculated to obtain the initial semantic similarity between different modality feature vectors. Security modulation of the initial semantic similarity is performed based on the security sensitivity weight coefficient. The security sensitivity weight coefficients corresponding to different modal feature vectors are multiplied with the initial semantic similarity to obtain the security-modulated semantic similarity. Each modal feature vector in the set of feature vectors with security labels is used as a node in the cross-modal association graph. Edge connections are established between different modal feature vector pairs whose semantic similarity after security modulation exceeds a preset similarity threshold. The semantic similarity after security modulation is used as the weight value of the edge connection to generate the cross-modal association graph.

3. The method according to claim 1, characterized in that, In the cross-modal association graph, the feature vectors with security labels are used as sensitive source nodes. The neighborhood node set and modal distribution characteristics of each sensitive source node are calculated. Based on the neighborhood node set and modal distribution characteristics, the sensitive source influence index is determined, including: In the cross-modal association graph, nodes with security-labeled feature vectors whose security label level exceeds the preset security level threshold are extracted as sensitive source nodes. For each sensitive source node, multi-hop graph expansion is performed along the edge connection in descending order of weight value. The set of nodes that can be reached by each sensitive source node within the preset expansion steps is obtained as the neighborhood node set. Calculate the shortest path length between each node in the neighborhood node set and the sensitive source node, and use the reciprocal of the shortest path length as the distance decay factor. Assign the distance decay factor as the node weight to each node in the neighborhood node set. The sum of node weights for each modality type in the neighborhood node set is counted, and the proportion of the sum of node weights for each modality type to the total weight of the neighborhood node set is calculated as the modality distribution feature. The information entropy of the modality distribution feature is then calculated to obtain the modality distribution uniformity value. The total weight of the neighborhood node set is used as the weighted diffusion range value. The weighted diffusion range value is multiplied by the modal distribution uniformity value to obtain the sensitive source influence index.

4. The method according to claim 1, characterized in that, Nodes associated with multiple sensitive source nodes in the cross-modal correlation graph are identified as risk convergence nodes. The security type of the sensitive source nodes associated with each risk convergence node is calculated, and the risk convergence coefficient is determined based on the security type, including: Traverse all nodes in the cross-modal association graph, count the number of sensitive source nodes directly associated with each node, and identify nodes whose number of directly associated sensitive source nodes exceeds a preset aggregation threshold as risk aggregation nodes; Extract the security identification level information of each sensitive source node directly associated with the risk aggregation node, classify each sensitive source node into security types based on the security identification level information, and obtain the security type set of the risk aggregation node; Traverse the security types in the security type set, determine the conflict between any two security types according to the preset security type conflict rules, and construct a conflict topology graph by using the security type pairs with conflicting relationships as edges. The nodes of the conflict topology graph are the security types in the security type set. Calculate the degree of each node in the conflict topology graph, where the degree is the number of edges connected to the current node. Mark the node with the largest degree as the core conflict node and obtain the security type corresponding to the core conflict node as the core conflict type. The number of sensitive source nodes whose security type belongs to the core conflict type among the directly associated sensitive source nodes of the statistical risk aggregation node is used as the risk aggregation coefficient.

5. The method according to claim 1, characterized in that, A risk propagation matrix is ​​constructed based on the influence index of sensitive sources and the risk convergence coefficient. The risk propagation matrix is ​​then subjected to eigenvalue decomposition to identify the dominant risk propagation direction, generating a risk propagation vector including: Traverse the propagation path from the sensitive source node to the risk convergence node in the cross-modal association graph, extract the modal types of adjacent nodes on the propagation path, determine whether there is a propagation blocking relationship between the modal types of adjacent nodes according to the preset modal compatibility rules, and count the number of adjacent node pairs with a propagation blocking relationship in the propagation path as the number of propagation blocking points. A propagation path with zero propagation blockade points is marked as a valid propagation path; Extract the sensitive source influence index of sensitive source nodes and the risk convergence coefficient of risk convergence nodes corresponding to effective propagation paths. For each node pair from a sensitive source node to each risk convergence node, count the number of effective propagation paths connecting the node pairs as the path connectivity. A risk propagation matrix is ​​constructed based on the sensitive source influence index corresponding to the sensitive source node, the risk convergence coefficient corresponding to the risk convergence node, and the path connectivity of the node pair. The row index of the risk propagation matrix is ​​the sensitive source node, and the column index is the risk convergence node. For each node pair, the corresponding sensitive source influence index, risk convergence coefficient, and path connectivity are used as input parameters to calculate the matrix element value. The risk propagation matrix is ​​decomposed into eigenvalues, and the eigenvector corresponding to the largest eigenvalue is used as the risk propagation vector.

6. A multimodal data security identification and intelligent classification processing system, used to implement the method as described in any one of claims 1-5, characterized in that, include: The multimodal labeling unit is used to acquire heterogeneous source multimodal data and extract feature vectors of each modality, label each feature vector with security attributes, and generate a set of feature vectors with security labels. Cross-modal building blocks are used to construct cross-modal association graphs based on the semantic similarity between feature vectors of different modalities in a set of feature vectors with security labels; Sensitive source refers to the unit, which is used in the cross-modal association graph to use feature vectors with security labels as sensitive source nodes, calculate the set of neighborhood nodes and modal distribution characteristics of each sensitive source node, and determine the sensitive source influence index based on the set of neighborhood nodes and modal distribution characteristics; The risk aggregation unit is used to identify nodes that are associated with multiple sensitive source nodes in the cross-modal association graph as risk aggregation nodes, count the security type of the sensitive source nodes associated with each risk aggregation node, and determine the risk aggregation coefficient based on the security type. The risk propagation unit is used to construct a risk propagation matrix based on the influence index of sensitive sources and the risk convergence coefficient, perform eigenvalue decomposition on the risk propagation matrix to identify the dominant risk propagation direction, and generate a risk propagation vector. The fusion classification unit is used to calculate the directional decay factor of each edge in the cross-modal association graph based on the risk propagation vector, adjust the edge weights based on the directional decay factor, generate an adjusted cross-modal association graph, perform feature fusion based on the adjusted cross-modal association graph, and perform classification processing.

7. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images