A multi-technology integrated method and system for hierarchical classification of multi-source data

Abstract

This invention discloses a multi-technology integrated method and system for hierarchical classification of multi-source data, comprising the following steps: S1, determining the data source, collecting multi-source data, and preprocessing it; S2, using CDC technology for monitoring and extracting the monitored target data; S3, extracting features from the target data and introducing the EMAP method to perform nonlinear cross-projection on the multimodal feature set; S4, inputting the high-dimensional feature vector into a hybrid classification network to perform principal component subset space compression and unsupervised clustering; S5, using the FP-Growth algorithm to mine high-frequency attribute items and construct privacy attribute sets and non-privacy attribute sets; S6, calculating the weighted privacy degree of each privacy cluster and performing multi-level privacy level division through a hierarchical mapping network; S7, encrypting and de-identifying the data under each label and outputting the visualization. This invention improves the intelligence and security of multi-source data classification and privacy hierarchical classification.

Description

Technical Field

[0001] This invention relates to the field of big data processing and fusion technology, and in particular to a multi-technology fusion method and system for hierarchical classification of multi-source data. Background Technology

[0002] With the deepening of digitalization, various organizations generate massive amounts of diverse multi-source data during information collection, data flow, and business operations. This data is widely distributed across traditional relational databases, big data distributed platforms, IoT sensor networks, and third-party interface services, exhibiting a complex mix of structured, semi-structured, and unstructured data. Especially in key areas such as smart government, financial risk control, healthcare, and the industrial internet, multi-source data often exhibits high sensitivity, high coupling, and strong business relevance, covering multiple modalities including text, tables, images, videos, and protocol interface response results. Against this backdrop, effectively integrating, accurately classifying, and appropriately grading this heterogeneous data has become a core challenge in information security management, privacy protection, and data governance.

[0003] Existing data classification and grading methods typically rely on rule engines and expert experience for attribute definition and level determination. While these methods are feasible for scenarios with well-defined structures and relatively standardized data, they fall short in terms of generalization ability, real-time performance, and automation as data scales grow exponentially and data sources diversify. Traditional methods often employ static feature tables, fixed matching rules, and manually encoded models for coarse-grained data processing, making them unable to handle new data formats and the high-dimensional dynamic changes of modal fusion inputs. Furthermore, many existing methods have significant limitations when processing unstructured data such as images and videos, primarily due to a lack of unified standards for feature extraction, a lack of cross-modeling mechanisms between modalities, and a lack of contextual semantic understanding in the data fusion stage, ultimately affecting the accuracy of data classification and the scientific rigor of grading.

[0004] In the feature extraction stage, most current technologies employ a single-modal vector modeling strategy, failing to fully leverage the complementarity between multimodal data. Especially when dealing with business data generated from a mixture of video, images, and text, existing methods often require constructing multiple models to process different modalities separately, then fusing the outputs through hard concatenation, lacking an effective expression of the nonlinear interactions between modal features. Furthermore, for assessing the privacy sensitivity of internal data attributes, existing methods mostly rely on fixed rules or manual annotation for subjective grading, failing to establish an adaptive grading mechanism based on objective statistics and multi-layered semantic understanding. This makes the processing of different data in different scenarios inflexible, posing a risk of grading distortion and misjudgment.

[0005] In the area of ​​privacy information identification, although some studies have attempted to introduce unsupervised learning algorithms such as principal component analysis and K-means to perform dimensionality reduction and clustering of features, these methods generally suffer from strong dependence on the number of clusters, ambiguous cluster boundaries, and poor interpretability of results. This makes it difficult to form a stable and accurate classification model during data classification, especially when identifying high-frequency sensitive attributes. Furthermore, for the problem of data grading after classification, existing solutions often use manually defined mapping tables for grading, ignoring the continuous variation of data features between different privacy levels, lacking quantitative indicators, and failing to achieve fine-grained, highly discriminative graded output.

[0006] Furthermore, existing classification and grading methods lack a unified scheduling architecture for model fusion mechanisms and upstream and downstream data processing, resulting in fragmented information transmission between models and a lack of end-to-end data processing loops. This fragmented and static processing approach makes it difficult for the system to respond quickly and self-correct when faced with dynamic business scenario changes, data structure adjustments, and fluctuations in content sensitivity levels. Simultaneously, existing methods often employ a one-size-fits-all full-field masking strategy in the data anonymization and result display stages, failing to combine the actual classification and grading results of the data to perform targeted encryption and visualization configurations for different privacy levels. This significantly impacts the subsequent usability of data assets and the sophistication of business processing.

[0007] Therefore, how to provide a multi-technology integrated method and system for hierarchical classification of multi-source data is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] One objective of this invention is to propose a multi-technology integrated method and system for hierarchical classification of multi-source data. This invention fully integrates key algorithms such as CDC listening extraction, EMAP multimodal projection, FP-Growth frequent item mining, information entropy quantification and hierarchical classification, and residual mapping network. It describes in detail the entire process from heterogeneous data acquisition, nonlinear feature fusion, privacy attribute identification, hierarchical discrimination to de-identified display, and has the advantages of high automation, fine hierarchical granularity, high classification accuracy and strong modality fusion capability.

[0009] A multi-technology fusion method for hierarchical classification of multi-source data according to an embodiment of the present invention includes the following steps:

[0010] S1. Determine the data source, collect data from multiple sources, and perform preprocessing;

[0011] S2. Use CDC technology to monitor data, set data monitoring rules, and extract the monitored target data from the preprocessed multi-source data and store it in the data warehouse.

[0012] S3. Extract features from the target data, construct a multimodal feature set, and introduce the EMAP method to perform nonlinear cross-projection on the multimodal feature set to obtain a high-dimensional feature vector.

[0013] S4. Input the high-dimensional feature vector into the hybrid classification network, perform principal component subspace compression and unsupervised clustering, and obtain the cluster labels and corresponding data attribute partitioning results;

[0014] S5. Employ the FP-Growth algorithm to mine high-frequency attribute items, construct a privacy attribute set and a non-privacy attribute set, and perform subset partitioning on the privacy attribute set to obtain multiple disjoint privacy cluster sets;

[0015] S6. Introduce a hierarchical discriminant function based on maximum entropy discrete quantization, calculate the weighted privacy degree of each privacy cluster, and perform multi-level privacy level division through a hierarchical mapping network to generate multi-level classification labels;

[0016] S7. Use a hash-based desensitization algorithm to encrypt and desensitize the data under each label, and store the encrypted data and labels in the visualization output directory to achieve desensitized presentation of classification results and hierarchical labels.

[0017] Optionally, the data source includes traditional databases, big data platforms such as Hadoop, wireless sensors, and third-party interfaces. The multi-source data includes text data, tabular data, image data, video data, and data returned by interfaces. The preprocessing includes integrity checks, consistency checks, outlier removal, noisy data cleaning, data format conversion, and data standardization.

[0018] Optionally, the hybrid classification network is constructed based on PCA dimensionality reduction and K-means clustering. The PCA dimensionality reduction compresses the input high-dimensional feature vector by using the maximum variance preservation criterion to generate a low-dimensional principal component representation. The K-means clustering calculates the Euclidean distance between samples based on the low-dimensional principal component representation and introduces a dynamic cluster number estimation mechanism. The number of clusters k is dynamically determined by the optimal silhouette coefficient criterion to improve the adaptive ability of clustering and the stability of cluster boundaries.

[0019] Optionally, the hierarchical mapping network is constructed based on residual connection MLP, and adopts a dual-channel input method to receive the privacy cluster feature vector and the corresponding weighted entropy value respectively. It extracts deep-level feature information through a cascaded attention mechanism, and introduces a multi-scale gating unit to adjust the segmented response for different weighted privacy ranges, so as to achieve adaptive mapping output for multiple privacy levels.

[0020] Optionally, S2 specifically includes:

[0021] S21. Set the monitoring strategy parameters according to the sampling requirements, with the sampling quantity being... The sampling interval is The number of concurrent samples is The number of samples Calculated using the following formula:

[0022] ;

[0023] in, This represents the quantile value corresponding to the confidence level on a standard normal distribution. This represents the estimated proportion of the currently monitored target data in the sample space. Indicates the allowable error range;

[0024] S22, Based on the number of samples Sampling interval With sampling concurrency Construct monitoring rules based on change data capture, and extract change content distributed in multi-source data at a set frequency;

[0025] S23. Perform format standardization on the monitored and extracted target data, convert text data and tabular data into a unified CSV format, call OpenCV functions to extract frames from video data to generate image data, and align all extracted data according to the field structure.

[0026] S24. Write the standardized target data into the data warehouse to complete the unified storage of data and generate a multi-source monitoring data set. Used for feature modeling and fusion analysis, among which Indicates based on strategy parameters , , The dataset after extraction and standardization.

[0027] Optionally, S3 specifically includes:

[0028] S31, Collect multi-source monitoring data Based on data mode, it is divided into There are several modal sets, each consisting of a text modal set. Image modal set Table Modal Sets Video modal set With interface modal set ;

[0029] S32. Perform feature mapping operation on each modality set to obtain the corresponding modality feature representation vector set. ,in , respectively, represent the vector sets of the five modalities in a unified feature space;

[0030] S33, Set modal features By inputting the empirical multimodal additive projection function, a cross-modal nonlinear high-order feature combination expression is constructed to obtain a high-dimensional feature vector:

[0031] ;

[0032] in, This represents the high-dimensional feature vector after fusion. This indicates the number of modes, with a value of 5. Indicates the first The original feature vector of the modality, , They represent the first The trainable weight matrix of the modality during the linear transformation process. , They represent the first Modal bias terms, , These are modal weighting coefficients, used to adjust the dominance of each modal feature in the fusion process. , These represent the nonlinear activation function and the hyperbolic tangent function, respectively. Indicates empirical perturbation variables Normalized noise projection is used to enhance feature robustness. This represents the perturbation influence factor, which adjusts the intensity of the perturbation projection's influence on the overall fusion characteristics.

[0033] Optionally, S4 specifically includes:

[0034] S41. Transform high-dimensional feature vectors Inputting the hybrid classification network, a principal component subset space compression operation is performed to construct a low-dimensional vector set. The compression operation adopts the maximum variance preservation criterion, selecting vectors whose cumulative variance contribution rate is not less than a set threshold. The former One principal component;

[0035] S42. Perform unsupervised clustering analysis on low-dimensional vector sets, using a dynamic clustering method based on Euclidean distance metric. Calculate the distance between points in the sample space and construct a cluster label set. The total number of clusters is determined using the optimal silhouette coefficient criterion. ;

[0036] S43. Assume each sample corresponds one-to-one with a low-dimensional vector, and perform a clustering prediction process to obtain a data attribute label set. The compression operation and the clustering prediction process are combined as follows:

[0037] ;

[0038] in, Represents a set of data attribute labels. This represents the high-dimensional feature vector after fusion. Indicates will Perform principal component analysis and retain a variance proportion of at least [percentage missing]. The low-dimensional representation, Represents the clustering function. Indicates the number of candidate clusters. This represents the total number of clusters. Indicates the first The average feature distance of samples within a cluster, This represents the overall average distance across all clusters. Indicates the first The standard deviation of the feature distance between samples within a cluster This represents a stability adjustment factor to prevent the denominator from approaching zero. Indicates the first The contour coefficients corresponding to each cluster Represents the number of all candidate clusters The value with the largest weighted index is selected as the final number of clusters.

[0039] S44. Transfer the data attribute label set Cluster label set Pairing and combining data generates a clustering label mapping dictionary for privacy attribute partitioning.

[0040] Optionally, S5 specifically includes:

[0041] S51. Input the data set corresponding to each cluster in the cluster label mapping dictionary into the frequent item mining function, execute the FP-Growth algorithm on each cluster data, extract the corresponding high-frequency attribute item set, and form a frequent attribute set.

[0042] S52. Construct an attribute distribution matrix based on frequent attribute sets. The rows represent clusters, the columns represent attribute fields, and the matrix elements are... Represents attributes In the The normalized frequency values ​​appearing in each cluster;

[0043] S53. Calculate the distribution entropy of all attributes across different clusters, and convert the attribute distribution matrix... Perform privacy attribute determination and classify the attributes into privacy attribute sets according to the privacy determination threshold. With non-privacy attribute set ;

[0044] S54, Set privacy attributes The attribute vectors in the data are re-aggregated based on clustering labels to form a set of disjoint privacy clusters. Each privacy cluster represents a set of sensitive information with similar structures and isomorphic attributes.

[0045] ;

[0046] in, Describes a set of disjoint privacy clusters. This represents the total number of clusters. Indicates the first Data sets in each cluster Indicates the first The attribute vector of each privacy cluster, Represents a sample data vector. Indicates the total number of attribute dimensions. This represents a small positive constant to prevent the logarithmic function from having zero input. Represents attributes The cross-cluster distribution entropy is used to represent the uncertainty of an attribute across all clusters. Indicates the threshold for privacy judgment. Represents the logarithmic function. Indicates belonging to, The symbol representing intersection in algebraic operations.

[0047] Optionally, S6 specifically includes:

[0048] S61, Set up privacy clusters Each privacy cluster is represented as a set of attribute vectors. The information entropy value of the set of attribute vectors is calculated and compared with the maximum theoretical information entropy. Perform the difference operation and normalization operation to obtain the normalized information entropy value;

[0049] S62. Align and concatenate the attribute vectors of the privacy clusters with their corresponding normalized information entropy values, and input them into a hierarchical mapping network to perform multi-level privacy level division, outputting a hierarchical label set:

[0050] ;

[0051] in, This is a multi-level classification label set, representing the level label for each privacy cluster. Represents a hierarchical mapping function. Indicates the total number of privacy clusters. Indicates the first The attribute vector of each privacy cluster, Indicates the first The information entropy value of a privacy cluster This represents a minimal constant that prevents the denominator of the logarithmic function from being zero. This represents the gated control function, where the first term is linear projection and the second term is weighted control. , They represent the first The weight matrix and bias terms of each privacy cluster Indicates the first The characteristic activation coefficients of each privacy cluster This represents the residual mapping adjustment coefficient. Represents attribute vectors based on residual structure Nonlinear compensation processing, This represents the activation coefficient of the entropy mapping.

[0052] A multi-technology fusion multi-source data hierarchical classification system according to an embodiment of the present invention includes:

[0053] The data source integration module is used to determine the data source, collect data from multiple sources, and perform preprocessing.

[0054] The data monitoring module is used to monitor data using CDC technology, set data monitoring rules, and extract the monitored target data from the preprocessed multi-source data and store it in the data warehouse.

[0055] The feature projection module is used to extract features from the target data, construct a multimodal feature set, and introduce the EMAP method to perform nonlinear cross projection on the multimodal feature set to obtain a high-dimensional feature vector.

[0056] The compressed clustering module is used to input high-dimensional feature vectors into the hybrid classification network, perform principal component subset space compression and unsupervised clustering, and obtain cluster labels and corresponding data attribute partitioning results;

[0057] The mining and partitioning module is used to mine high-frequency attribute items using the FP-Growth algorithm, construct a privacy attribute set and a non-privacy attribute set, and perform subset partitioning on the privacy attribute set to obtain multiple disjoint privacy cluster sets.

[0058] The label calculation module is used to introduce a hierarchical discriminant function based on maximum entropy discrete quantization, calculate the weighted privacy degree of each privacy cluster, and perform multi-level privacy level division through a hierarchical mapping network to generate multi-level classification labels;

[0059] The encrypted visualization module is used to encrypt and de-identify the data under each category label using a hash de-identification algorithm, and stores the encrypted data and labels in the visualization output directory to achieve de-identified presentation of classification results and hierarchical labels.

[0060] The beneficial effects of this invention are:

[0061] First, this invention addresses the problems of traditional data acquisition methods' inability to capture data in real time and inconsistent multimodal data structures by introducing CDC (Change Data Capture) technology and a multi-source heterogeneous data monitoring strategy. By combining unified preprocessing and format standardization operations for five typical modalities—text, image, table, video, and interface data—an end-to-end data source fusion channel is constructed, providing a stable and standardized basic input environment for subsequent feature extraction and modeling.

[0062] Secondly, this invention constructs a nonlinear fusion model based on Empirical Multimodal Additive Function Projection (EMAP), significantly improving the feature collaborative modeling capability among multimodal data. Combining PCA principal component analysis and K-means dynamic clustering algorithm, it achieves compressed representation and adaptive clustering of high-dimensional features, enhancing the system's ability to classify and identify complex heterogeneous information. Simultaneously, by introducing the FP-Growth algorithm to mine frequent attribute terms and establish privacy attribute identification rules, and combining the maximum entropy difference quantization strategy and residual gating network to complete fine-grained multi-level hierarchical discrimination, it significantly improves the accuracy and dynamic response capability of data security level classification.

[0063] Finally, in the result processing stage, this invention constructs a data anonymization and visualization mechanism based on a hash algorithm, realizing differentiated encryption and secure output of data with different privacy levels, effectively improving the business usability and compliant display capabilities of anonymized data. The overall process achieves a closed loop from data monitoring and collection, feature extraction, classification and grading to encrypted presentation. Compared with existing technologies, this invention has significant improvements in real-time performance, accuracy, automation, interpretability, and system scalability, meeting the needs for secure identification and governance of multimodal heterogeneous data. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0065] Figure 1 This is a flowchart of a multi-technology fusion multi-source data hierarchical classification method proposed in this invention;

[0066] Figure 2 This is a flowchart illustrating the feature extraction and fusion mapping process of a multi-source data hierarchical classification method integrating multiple technologies proposed in this invention.

[0067] Figure 3 This is a module structure diagram of a multi-source data hierarchical classification system that integrates multiple technologies proposed in this invention. Detailed Implementation

[0068] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0069] refer to Figure 1-2 A multi-technology integrated method for hierarchical classification of multi-source data includes the following steps:

[0070] S1. Determine the data source, collect data from multiple sources, and perform preprocessing;

[0071] S2. Use CDC technology to monitor data, set data monitoring rules, and extract the monitored target data from the preprocessed multi-source data and store it in the data warehouse.

[0072] S3. Extract features from the target data, construct a multimodal feature set, and introduce the EMAP method to perform nonlinear cross-projection on the multimodal feature set to obtain a high-dimensional feature vector.

[0073] S4. Input the high-dimensional feature vector into the hybrid classification network, perform principal component subspace compression and unsupervised clustering, and obtain the cluster labels and corresponding data attribute partitioning results;

[0074] S5. Employ the FP-Growth algorithm to mine high-frequency attribute items, construct a privacy attribute set and a non-privacy attribute set, and perform subset partitioning on the privacy attribute set to obtain multiple disjoint privacy cluster sets;

[0075] S6. Introduce a hierarchical discriminant function based on maximum entropy discrete quantization, calculate the weighted privacy degree of each privacy cluster, and perform multi-level privacy level division through a hierarchical mapping network to generate multi-level classification labels;

[0076] S7. Use a hash-based desensitization algorithm to encrypt and desensitize the data under each label, and store the encrypted data and labels in the visualization output directory to achieve desensitized presentation of classification results and hierarchical labels.

[0077] This invention provides a complete multi-technology integrated method for hierarchical classification of multi-source data. It constructs a closed-loop system covering the entire process from multi-source data acquisition, heterogeneous data preprocessing, multi-modal feature fusion, classification and clustering recognition to hierarchical desensitization and presentation. It has the advantages of high automation, unified processing flow, and strong adaptability, which significantly improves the efficiency of data governance and the accuracy of security level identification in complex environments.

[0078] In this embodiment, the data sources include traditional databases, big data platforms such as Hadoop, wireless sensors, and third-party interfaces. The multi-source data includes text data, tabular data, image data, video data, and interface return data. The preprocessing includes integrity checks, consistency checks, outlier removal, noisy data cleaning, data format conversion, and data standardization.

[0079] This invention ensures the consistency of fusion processing of structured, semi-structured, and unstructured data by clearly specifying the supported multi-source data types and their preprocessing mechanisms. This effectively improves the standardization of data integration and the quality control capabilities of the preprocessing stage, providing a clear, clean, and high-quality input data foundation for subsequent modeling.

[0080] In this embodiment, the hybrid classification network is constructed based on PCA dimensionality reduction and K-means clustering. PCA dimensionality reduction compresses the input high-dimensional feature vector by using the maximum variance preservation criterion to generate a low-dimensional principal component representation. K-means clustering calculates the Euclidean distance between samples based on the low-dimensional principal component representation and introduces a dynamic cluster number estimation mechanism. The number of clusters k is dynamically determined by the optimal silhouette coefficient criterion to improve the adaptive ability of clustering and the stability of cluster boundaries.

[0081] This invention introduces a combined mechanism of PCA principal component compression and K-means dynamic clustering into a hybrid classification network, which significantly improves the compression effect and clustering accuracy of high-dimensional features, has strong adaptability and clustering stability, and effectively solves the problems of strong cluster number dependence and unclear classification boundaries in traditional clustering methods.

[0082] In this embodiment, the hierarchical mapping network is constructed based on the residual connection MLP. It adopts a dual-channel input method to receive the privacy cluster feature vector and the corresponding weighted entropy value respectively. It extracts deep-level feature information through a cascaded attention mechanism and introduces a multi-scale gating unit to adjust the segmented response to different weighted privacy ranges, thereby realizing adaptive mapping output for multiple privacy levels.

[0083] This invention designs a hierarchical mapping network based on residual connections, combining dual-channel input, cascaded attention mechanism and multi-scale gating structure to achieve accurate modeling and adaptive segmented response of weighted privacy information, thereby improving the accuracy, continuity and interpretability of privacy level labels.

[0084] In this embodiment, S2 specifically includes:

[0085] S21. Set the monitoring strategy parameters according to the sampling requirements, with the sampling quantity being... The sampling interval is The number of concurrent samples is The number of samples Calculated using the following formula:

[0086] ;

[0087] in, This represents the quantile value corresponding to the confidence level on a standard normal distribution. This represents the estimated proportion of the currently monitored target data in the sample space. Indicates the allowable error range;

[0088] S22, Based on the number of samples Sampling interval With sampling concurrency Construct monitoring rules based on change data capture, and extract change content distributed in multi-source data at a set frequency;

[0089] S23. Perform format standardization on the monitored and extracted target data, convert text data and tabular data into a unified CSV format, call OpenCV functions to extract frames from video data to generate image data, and align all extracted data according to the field structure.

[0090] S24. Write the standardized target data into the data warehouse to complete the unified storage of data and generate a multi-source monitoring data set. Used for feature modeling and fusion analysis, among which Indicates based on strategy parameters , , The dataset after extraction and standardization.

[0091] This invention employs CDC technology during data monitoring and sets sampling rule parameters through a mathematical sampling model. It introduces quantitative calculations of confidence quantiles, error tolerances, and sample proportions, thereby achieving efficient monitoring and precise control of real-time data changes and improving the representativeness and processing efficiency of the monitored data.

[0092] In this embodiment, S3 specifically includes:

[0093] S31, Collect multi-source monitoring data Based on data mode, it is divided into There are several modal sets, each consisting of a text modal set. Image modal set Table Modal Sets Video modal set With interface modal set ;

[0094] S32. Perform feature mapping operation on each modality set to obtain the corresponding modality feature representation vector set. ,in , respectively, represent the vector sets of the five modalities in a unified feature space;

[0095] S33, Set modal features By inputting the empirical multimodal additive projection function, a cross-modal nonlinear high-order feature combination expression is constructed to obtain a high-dimensional feature vector:

[0096] ;

[0097] in, This represents the high-dimensional feature vector after fusion. This indicates the number of modes, with a value of 5. Indicates the first The original feature vector of the modality, , They represent the first The trainable weight matrix of the modality during the linear transformation process. , They represent the first Modal bias terms, , These are modal weighting coefficients, used to adjust the dominance of each modal feature in the fusion process. , These represent the nonlinear activation function and the hyperbolic tangent function, respectively. Indicates empirical perturbation variables Normalized noise projection is used to enhance feature robustness. This represents the perturbation influence factor, which adjusts the intensity of the perturbation projection's influence on the overall fusion characteristics.

[0098] This invention utilizes the EMAP empirical multimodal additive function to perform nonlinear cross-fusion of multimodal features such as text, images, and videos. For the first time, it realizes the additivity and learnable representation of multimodal data in a unified high-dimensional space, significantly enhancing the model's ability to understand the semantic relationships between heterogeneous data.

[0099] In this embodiment, S4 specifically includes:

[0100] S41. Transform high-dimensional feature vectors Inputting the hybrid classification network, a principal component subset space compression operation is performed to construct a low-dimensional vector set. The compression operation adopts the maximum variance preservation criterion, selecting vectors whose cumulative variance contribution rate is not less than a set threshold. The former One principal component;

[0101] S42. Perform unsupervised clustering analysis on low-dimensional vector sets, using a dynamic clustering method based on Euclidean distance metric. Calculate the distance between points in the sample space and construct a cluster label set. The total number of clusters is determined using the optimal silhouette coefficient criterion. ;

[0102] S43. Assume each sample corresponds one-to-one with a low-dimensional vector, and perform a clustering prediction process to obtain a data attribute label set. The compression operation and the clustering prediction process are combined as follows:

[0103] ;

[0104] in, Represents a set of data attribute labels. This represents the high-dimensional feature vector after fusion. Indicates will Perform principal component analysis and retain a variance proportion of at least [percentage missing]. The low-dimensional representation, Represents the clustering function. Indicates the number of candidate clusters. This represents the total number of clusters. Indicates the first The average feature distance of samples within a cluster, This represents the overall average distance across all clusters. Indicates the first The standard deviation of the feature distance between samples within a cluster This represents a stability adjustment factor to prevent the denominator from approaching zero. Indicates the first The contour coefficients corresponding to each cluster Represents the number of all candidate clusters The value with the largest weighted index is selected as the final number of clusters.

[0105] S44. Transfer the data attribute label set Cluster label set Pairing and combining data generates a clustering label mapping dictionary for privacy attribute partitioning.

[0106] This invention constructs a cluster label set by high-dimensional feature principal component compression and cluster optimization function, and uses the silhouette coefficient as the basis for dynamic adjustment of the number of clusters, so that the classification results have stronger generalization and expansion capabilities while maintaining accuracy, and effectively adapt to high-dimensional irregular data distribution environment.

[0107] In this embodiment, S5 specifically includes:

[0108] S51. Input the data set corresponding to each cluster in the cluster label mapping dictionary into the frequent item mining function, execute the FP-Growth algorithm on each cluster data, extract the corresponding high-frequency attribute item set, and form a frequent attribute set.

[0109] S52. Construct an attribute distribution matrix based on frequent attribute sets. The rows represent clusters, the columns represent attribute fields, and the matrix elements are... Represents attributes In the The normalized frequency values ​​appearing in each cluster;

[0110] S53. Calculate the distribution entropy of all attributes across different clusters, and convert the attribute distribution matrix... Perform privacy attribute determination and classify the attributes into privacy attribute sets according to the privacy determination threshold. With non-privacy attribute set ;

[0111] S54, Set privacy attributes The attribute vectors in the data are re-aggregated based on clustering labels to form a set of disjoint privacy clusters. Each privacy cluster represents a set of sensitive information with similar structures and isomorphic attributes.

[0112] ;

[0113] in, Describes a set of disjoint privacy clusters. This represents the total number of clusters. Indicates the first Data sets in each cluster Indicates the first The attribute vector of each privacy cluster, Represents a sample data vector. Indicates the total number of attribute dimensions. This represents a small positive constant to prevent the logarithmic function from having zero input. Represents attributes The cross-cluster distribution entropy is used to represent the uncertainty of an attribute across all clusters. Indicates the threshold for privacy judgment. Represents the logarithmic function. Indicates belonging to, The symbol representing intersection in algebraic operations.

[0114] This invention introduces the FP-Growth algorithm to identify frequent items, constructs a privacy attribute set, and combines attribute distribution entropy to construct a privacy cluster. This improves the fine-grained level and objectivity of sensitive data identification, overcomes the problem of traditional static rule-based classification methods relying on expert experience, and has stronger data self-interpretation capabilities.

[0115] In this embodiment, S6 specifically includes:

[0116] S61, Set up privacy clusters Each privacy cluster is represented as a set of attribute vectors. The information entropy value of the set of attribute vectors is calculated and compared with the maximum theoretical information entropy. Perform the difference operation and normalization operation to obtain the normalized information entropy value;

[0117] S62. Align and concatenate the attribute vectors of the privacy clusters with their corresponding normalized information entropy values, and input them into a hierarchical mapping network to perform multi-level privacy level division, outputting a hierarchical label set:

[0118] ;

[0119] in, This is a multi-level classification label set, representing the level label for each privacy cluster. Represents a hierarchical mapping function. Indicates the total number of privacy clusters. Indicates the first The attribute vector of each privacy cluster, Indicates the first The information entropy value of a privacy cluster This represents a minimal constant that prevents the denominator of the logarithmic function from being zero. This represents the gated control function, where the first term is linear projection and the second term is weighted control. , They represent the first The weight matrix and bias terms of each privacy cluster Indicates the first The characteristic activation coefficients of each privacy cluster This represents the residual mapping adjustment coefficient. Represents attribute vectors based on residual structure Nonlinear compensation processing, This represents the activation coefficient of the entropy mapping.

[0120] This invention introduces a normalized privacy entropy modeling mechanism based on the maximum entropy discrete quantization concept and integrates residual nonlinear structures to realize hierarchical response and nonlinear mapping output for privacy level determination, thereby improving the accurate representation of hierarchical labels and balancing classification strength and controllable interpretation.

[0121] refer to Figure 3 A multi-technology integrated multi-source data hierarchical classification system, comprising:

[0122] The data source integration module is used to determine the data source, collect data from multiple sources, and perform preprocessing.

[0123] The data monitoring module is used to monitor data using CDC technology, set data monitoring rules, and extract the monitored target data from the preprocessed multi-source data and store it in the data warehouse.

[0124] The feature projection module is used to extract features from the target data, construct a multimodal feature set, and introduce the EMAP method to perform nonlinear cross projection on the multimodal feature set to obtain a high-dimensional feature vector.

[0125] The compressed clustering module is used to input high-dimensional feature vectors into the hybrid classification network, perform principal component subset space compression and unsupervised clustering, and obtain cluster labels and corresponding data attribute partitioning results;

[0126] The mining and partitioning module is used to mine high-frequency attribute items using the FP-Growth algorithm, construct a privacy attribute set and a non-privacy attribute set, and perform subset partitioning on the privacy attribute set to obtain multiple disjoint privacy cluster sets.

[0127] The label calculation module is used to introduce a hierarchical discriminant function based on maximum entropy discrete quantization, calculate the weighted privacy degree of each privacy cluster, and perform multi-level privacy level division through a hierarchical mapping network to generate multi-level classification labels;

[0128] The encrypted visualization module is used to encrypt and de-identify the data under each category label using a hash de-identification algorithm, and stores the encrypted data and labels in the visualization output directory to achieve de-identified presentation of classification results and hierarchical labels.

[0129] This invention constructs a complete system architecture covering data acquisition, monitoring, fusion, classification, grading, encryption, and visualization output. The functional boundaries of each module are clear, the logic is closely coordinated, and it has good engineering feasibility and system scalability, significantly improving the security classification capability and management efficiency of multi-source data assets.

[0130] Example 1:

[0131] To verify the feasibility of this invention in practice, it was applied to a comprehensive data governance scenario. The data involved came from a database management platform, video surveillance platform, document management system, and multi-service interface gateway, covering typical multimodal data such as text reports, structured tables, surveillance video frames, and web interface response results. The main problem faced in this scenario is that, due to the complexity of data sources, inconsistent data formats, and significant differences in semantic levels between data, traditional data classification and grading methods are prone to omissions and misjudgments when processing such heterogeneous data. In particular, they lack objective measurement and automated processing capabilities in identifying and classifying privacy fields, leading to increased risks of privacy data leakage and high governance costs.

[0132] In this scenario, the system initially received 31,000 text data entries, 17,800 tabular data files, and 54,000 image data entries (generated from a video stream at 1 frame per second). The interface returned 22,000 records. During the data preprocessing stage, a monitoring rule was constructed using a CDC mechanism. Based on a 95% confidence level, a 5% tolerance, and an estimation ratio of 0.3, the required sampling quantity was calculated to be 967 entries. The sampling interval was set to 10 seconds, and the sampling concurrency was 32 threads. After extraction, the data was uniformly converted to CSV format and standardized, including field alignment, missing value imputation, and noise removal, before being written to the fused data warehouse.

[0133] In the feature processing stage, TF-IDF term frequency vectors and SIFT local feature descriptors are extracted through natural language segmentation and image feature extraction operations, respectively. The Empirical Multimodal Addition Function (EMAP) is then used to fuse vectors from five modalities: text, image, table, and video, generating a high-dimensional feature vector of 530 dimensions, forming a unified feature representation tensor. PCA compression is used to retain 95% of the variance information, ultimately resulting in a 92-dimensional principal component vector. The K-means clustering algorithm is then used to dynamically determine the number of clusters to be 7 under the optimal silhouette coefficient criterion. The system successfully outputs cluster labels based on the classification results and extracts approximately 4325 frequent itemsets using the FP-Growth algorithm. High-frequency sensitive fields such as "ID number," "address," "payment account," and "facial feature value" are automatically classified as privacy attributes.

[0134] After clustering based on privacy attributes, the data was further subdivided into 14 privacy clusters. Information entropy was calculated for each cluster, and a privacy level was determined by fusing a maximum entropy discrete function with a residual network. The final output level labels included three categories: Level 1 (highly sensitive), Level 2 (medium sensitive), and Level 3 (general). During the data anonymization phase, based on the different level labels, a hybrid encryption strategy using hash and Base64, along with field symbol masking, was employed to encrypt relevant fields. This resulted in a total of 68,400 encrypted data entries, all of which were stored in a visual management platform.

[0135] Experimental comparisons revealed that the privacy recognition accuracy using this invention increased from 83.2% to 94.6% compared to traditional methods, and the hierarchical consistency improved to 91.3%. Compared to rule-based schemes, processing time for multimodal data was reduced by 27%, and the classification error rate due to modality mismatch or feature conflict decreased from 13.5% to 4.9%. Furthermore, in the output visualization platform, the encryption strategies corresponding to different privacy levels effectively differentiate the visibility of sensitive fields, avoiding the data utilization degradation caused by a "one-size-fits-all" approach to desensitization.

[0136] The experiment shows that the present invention is highly feasible, highly automated, and has good scalability in multimodal data fusion, classification, and hierarchical processing, and has strong engineering application value in practical data governance.

[0137] Table 1 Comparison of Multimodal Data Fusion Classification and Hierarchical Processing Effects

[0138] Data types Original data volume Sample size Principal component dimension Cluster number High-frequency attribute items Privacy Clusters Classification accuracy Hierarchical consistency Data volume after anonymization Error rate decrease Text data 31,000 460 items 92-dimensional 7 1320 items 3 95.1% 91.6% 29,500 9.1% tabular data 17,800 210 items 92-dimensional 6 980 items 2 93.8% 90.2% 16,600 8.4% Image data 54,000 145 articles 92-dimensional 8 875 items 4 94.9% 92.4% 51,700 10.7% Video frame image 17300 frames 74 frames 92-dimensional 6 560 items 2 92.4% 89.7% 16100 frames 8.3% API returns data 22,000 78 articles 92-dimensional 7 590 items 3 95.3% 92.8% 20,600 7.9% Summary average 142,820 967 items 92-dimensional 6.8 4325 items 2.8 94.3% 91.3% 68,400 8.9%

[0139] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-technology integrated method for hierarchical classification of multi-source data, characterized in that, Includes the following steps: S1. Determine the data source, collect data from multiple sources, and perform preprocessing; The data sources include traditional databases, big data platforms such as Hadoop, wireless sensors, and third-party interfaces. The multi-source data includes text data, tabular data, image data, video data, and data returned by interfaces. The preprocessing includes integrity checks, consistency checks, outlier removal, noisy data cleaning, data format conversion, and data standardization. S2. Use CDC technology to monitor data, set data monitoring rules, and extract the monitored target data from the preprocessed multi-source data and store it in the data warehouse. S3. Extract features from the target data, construct a multimodal feature set, and introduce the EMAP method to perform nonlinear cross-projection on the multimodal feature set to obtain a high-dimensional feature vector. S4. Input the high-dimensional feature vector into the hybrid classification network, perform principal component subspace compression and unsupervised clustering, and obtain the cluster labels and corresponding data attribute partitioning results; The hybrid classification network is constructed based on PCA dimensionality reduction and K-means clustering. PCA dimensionality reduction compresses the input high-dimensional feature vector by using the maximum variance preservation criterion to generate a low-dimensional principal component representation. K-means clustering calculates the Euclidean distance between samples based on the low-dimensional principal component representation and introduces a dynamic cluster number estimation mechanism. The optimal determination of the silhouette coefficient dynamically determines the cluster number k value to improve the adaptive ability of clustering and the stability of cluster boundaries. S5. Employ the FP-Growth algorithm to mine high-frequency attribute items, construct a privacy attribute set and a non-privacy attribute set, and perform subset partitioning on the privacy attribute set to obtain multiple disjoint privacy cluster sets; S6. Introduce a hierarchical discriminant function based on maximum entropy discrete quantization, calculate the weighted privacy degree of each privacy cluster, and perform multi-level privacy level division through a hierarchical mapping network to generate multi-level classification labels; The hierarchical mapping network is constructed based on residual connection MLP. It adopts a dual-channel input method to receive the privacy cluster feature vector and the corresponding weighted entropy value respectively. It extracts deep-level feature information through a cascaded attention mechanism and introduces a multi-scale gating unit to adjust the segmented response for different weighted privacy ranges, so as to achieve adaptive mapping output for multiple privacy levels. S7. Use a hash-based desensitization algorithm to encrypt and desensitize the data under each label, and store the encrypted data and labels in the visualization output directory to achieve desensitized presentation of classification results and hierarchical labels.

2. The multi-technology fusion multi-source data hierarchical classification method according to claim 1, characterized in that, S2 specifically includes: S21. Set the monitoring strategy parameters according to the sampling requirements, with the sampling quantity being... The sampling interval is The number of concurrent samplings is The number of samples Calculated using the following formula: ； in, This represents the quantile value corresponding to the confidence level on a standard normal distribution. This represents the estimated proportion of the currently monitored target data in the sample space. Indicates the allowable error range; S22, Based on the number of samples Sampling interval With sampling concurrency Construct monitoring rules based on change data capture, and extract change content distributed in multi-source data at a set frequency; S23. Perform format standardization on the monitored and extracted target data, convert text data and tabular data into a unified CSV format, call OpenCV functions to extract frames from video data to generate image data, and align all extracted data according to the field structure. S24. Write the standardized target data into the data warehouse to complete the unified storage of data and generate a multi-source monitoring data set. Used for feature modeling and fusion analysis, among which Indicates based on strategy parameters , , The dataset after extraction and standardization.

3. The multi-technology fusion multi-source data hierarchical classification method according to claim 1, characterized in that, S3 specifically includes: S31, Collect multi-source monitoring data Based on data mode, it is divided into There are several modal sets, each consisting of a text modal set. Image modal set Table Modal Sets Video modal set With interface modal set ; S32. Perform feature mapping operation on each modality set to obtain the corresponding modality feature representation vector set. ,in , respectively, represent the vector sets of the five modalities in a unified feature space; S33, Set modal features By inputting the empirical multimodal additive projection function, a cross-modal nonlinear high-order feature combination expression is constructed to obtain a high-dimensional feature vector: ； in, This represents the high-dimensional feature vector after fusion. This indicates the number of modes, with a value of 5. Indicates the first The original feature vector of the modality, , They represent the first The trainable weight matrix of the modality during the linear transformation process. , They represent the first Modal bias terms, , These are modal weighting coefficients, used to adjust the dominance of each modal feature in the fusion process. , These represent the nonlinear activation function and the hyperbolic tangent function, respectively. Indicates empirical perturbation variables Normalized noise projection is used to enhance feature robustness. This represents the perturbation influence factor, which adjusts the intensity of the perturbation projection's influence on the overall fusion characteristics.

4. The multi-technology fusion multi-source data hierarchical classification method according to claim 1, characterized in that, S4 specifically includes: S41. Transform high-dimensional feature vectors Inputting the hybrid classification network, a principal component subset space compression operation is performed to construct a low-dimensional vector set. The compression operation adopts the maximum variance preservation criterion, selecting vectors whose cumulative variance contribution rate is not less than a set threshold. The former One principal component; S42. Perform unsupervised clustering analysis on low-dimensional vector sets, using a dynamic clustering method based on Euclidean distance metric. Calculate the distance between points in the sample space and construct a cluster label set. The total number of clusters is determined using the optimal silhouette coefficient criterion. ; S43. Assume each sample corresponds one-to-one with a low-dimensional vector, and perform a clustering prediction process to obtain a data attribute label set. The compression operation and the clustering prediction process are combined as follows: ； in, Represents a set of data attribute labels. This represents the high-dimensional feature vector after fusion. Indicates will Perform principal component analysis and retain a variance proportion of at least [percentage missing]. The low-dimensional representation, Represents the clustering function. Indicates the number of candidate clusters. This represents the total number of clusters. Indicates the first The average feature distance of samples within a cluster, This represents the overall average distance across all clusters. Indicates the first The standard deviation of the feature distance between samples within a cluster This represents a stability adjustment factor to prevent the denominator from approaching zero. Indicates the first The contour coefficients corresponding to each cluster Represents the number of all candidate clusters The value with the largest weighted index is selected as the final number of clusters. S44. Transfer the data attribute label set Cluster label set Pairing and combining data generates a clustering label mapping dictionary for privacy attribute partitioning.

5. The multi-technology fusion multi-source data hierarchical classification method according to claim 1, characterized in that, S5 specifically includes: S51. Input the data set corresponding to each cluster in the cluster label mapping dictionary into the frequent item mining function, execute the FP-Growth algorithm on each cluster data, extract the corresponding high-frequency attribute item set, and form a frequent attribute set. S52. Construct an attribute distribution matrix based on frequent attribute sets. The rows represent clusters, the columns represent attribute fields, and the matrix elements are... Representing attributes In the The normalized frequency values ​​appearing in each cluster; S53. Calculate the distribution entropy of all attributes across different clusters, and convert the attribute distribution matrix... Perform privacy attribute determination and classify the attributes into privacy attribute sets according to the privacy determination threshold. With non-privacy attribute set ; S54, Set privacy attributes The attribute vectors in the data are re-aggregated based on clustering labels to form a set of disjoint privacy clusters. Each privacy cluster represents a set of sensitive information with similar structures and isomorphic attributes. ； in, Describes a set of disjoint privacy clusters. This represents the total number of clusters. Indicates the first Data sets in each cluster Indicates the first The attribute vector of each privacy cluster, Represents a sample data vector. Indicates the total number of attribute dimensions. This represents a small positive constant to prevent the logarithmic function from having zero input. Representing attributes The cross-cluster distribution entropy is used to represent the uncertainty of an attribute across all clusters. Indicates the threshold for privacy judgment. Represents the logarithmic function. Indicates belonging to, The symbol representing intersection in algebraic operations.

6. The multi-technology fusion multi-source data hierarchical classification method according to claim 1, characterized in that, S6 specifically includes: S61, Set up privacy clusters Each privacy cluster is represented as a set of attribute vectors. The information entropy value of the set of attribute vectors is calculated and compared with the maximum theoretical information entropy. Perform the difference operation and normalization operation to obtain the normalized information entropy value; S62. Align and concatenate the attribute vectors of the privacy clusters with their corresponding normalized information entropy values, and input them into a hierarchical mapping network to perform multi-level privacy level division, outputting a hierarchical label set: ； in, This is a multi-level classification label set, representing the level label for each privacy cluster. Represents a hierarchical mapping function. Indicates the total number of privacy clusters. Indicates the first The attribute vector of each privacy cluster, Indicates the first The information entropy value of a privacy cluster This represents a minimal constant that prevents the denominator of the logarithmic function from being zero. This represents the gated control function, where the first term is linear projection and the second term is weighted control. , They represent the first The weight matrix and bias terms of each privacy cluster Indicates the first The characteristic activation coefficients of each privacy cluster This represents the residual mapping adjustment coefficient. Represents attribute vectors based on residual structure Nonlinear compensation processing, This represents the activation coefficient of the entropy mapping.

7. A multi-technology integrated multi-source data hierarchical classification system, implementing the multi-technology integrated multi-source data hierarchical classification method according to any one of claims 1 to 6, characterized in that, include: The data source integration module is used to determine the data source, collect data from multiple sources, and perform preprocessing. The data sources include traditional databases, big data platforms such as Hadoop, wireless sensors, and third-party interfaces. The multi-source data includes text data, tabular data, image data, video data, and data returned by interfaces. The preprocessing includes integrity checks, consistency checks, outlier removal, noisy data cleaning, data format conversion, and data standardization. The data monitoring module is used to monitor data using CDC technology, set data monitoring rules, and extract the monitored target data from the preprocessed multi-source data and store it in the data warehouse. The feature projection module is used to extract features from the target data, construct a multimodal feature set, and introduce the EMAP method to perform nonlinear cross projection on the multimodal feature set to obtain a high-dimensional feature vector. The compressed clustering module is used to input high-dimensional feature vectors into the hybrid classification network, perform principal component subset space compression and unsupervised clustering, and obtain cluster labels and corresponding data attribute partitioning results; The hybrid classification network is constructed based on PCA dimensionality reduction and K-means clustering. PCA dimensionality reduction compresses the input high-dimensional feature vector by using the maximum variance preservation criterion to generate a low-dimensional principal component representation. K-means clustering calculates the Euclidean distance between samples based on the low-dimensional principal component representation and introduces a dynamic cluster number estimation mechanism. The optimal determination of the silhouette coefficient dynamically determines the cluster number k value to improve the adaptive ability of clustering and the stability of cluster boundaries. The mining and partitioning module is used to mine high-frequency attribute items using the FP-Growth algorithm, construct a privacy attribute set and a non-privacy attribute set, and perform subset partitioning on the privacy attribute set to obtain multiple disjoint privacy cluster sets. The label calculation module is used to introduce a hierarchical discriminant function based on maximum entropy discrete quantization, calculate the weighted privacy degree of each privacy cluster, and perform multi-level privacy level division through a hierarchical mapping network to generate multi-level classification labels; The hierarchical mapping network is constructed based on residual connection MLP. It adopts a dual-channel input method to receive the privacy cluster feature vector and the corresponding weighted entropy value respectively. It extracts deep-level feature information through a cascaded attention mechanism and introduces a multi-scale gating unit to adjust the segmented response for different weighted privacy ranges, so as to achieve adaptive mapping output for multiple privacy levels. The encrypted visualization module is used to encrypt and de-identify the data under each category label using a hash de-identification algorithm, and stores the encrypted data and labels in the visualization output directory to achieve de-identified presentation of classification results and hierarchical labels.

Images